[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:12 dist-test-slave-3jql krb5kdc[10684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:12 dist-test-slave-3jql krb5kdc[10684](info): set up 2 sockets
Jun 20 18:34:12 dist-test-slave-3jql krb5kdc[10684](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:34:13.840190 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:13 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:13.966585 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42289
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:42289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:14.048470 10697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:14.048671 10697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:14.048694 10697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:14.049881 10697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:14.049924 10697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:14.049939 10697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:14.049952 10697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:14.049963 10697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:14.051553 10697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:42289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:14.051802 10697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:14.052026 10697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:14.057768 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.057819 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.057824 10697 server_base.cc:1034] running on GCE node
W20260620 18:34:14.057781 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.058240 10697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:14.058534 10697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:14.059725 10697 hybrid_clock.cc:648] HybridClock initialized: now 1781980454059692 us; error 42 us; skew 500 ppm
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:14.069543 10697 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:14.069989 10697 webserver.cc:460] Webserver started at http://127.10.109.126:35921/ using document root <none> and password file <none>
I20260620 18:34:14.070181 10697 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:14.070223 10697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:14.070308 10697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:14.071420 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "b1cd5a4afc7945db9962d3427295eae7"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "dffdafad36a981a1f1fdbe6fd5e9703e"
server_key_iv: "bb07fbd97f364d6b8509ae3ea791bd38"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.071723 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "b1cd5a4afc7945db9962d3427295eae7"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "dffdafad36a981a1f1fdbe6fd5e9703e"
server_key_iv: "bb07fbd97f364d6b8509ae3ea791bd38"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.072937 10697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:14.073709 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.073909 10697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:14.073989 10697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "b1cd5a4afc7945db9962d3427295eae7"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "dffdafad36a981a1f1fdbe6fd5e9703e"
server_key_iv: "bb07fbd97f364d6b8509ae3ea791bd38"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.074051 10697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:14.094053 10697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:14.096050 10697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:14.096159 10697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:14.100304 10697 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:42289
I20260620 18:34:14.100472 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:42289 every 8 connection(s)
I20260620 18:34:14.100700 10697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:14.101434 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 10697
I20260620 18:34:14.101563 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:14.101689 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:14.102226 10677 external_mini_cluster.cc:1428] Setting key f5d785871c83ab8bdbd79445ffc35a14
I20260620 18:34:14.104225 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: Bootstrap starting.
I20260620 18:34:14.104768 10765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:14.105047 10765 log.cc:824] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:14.105803 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: No bootstrap required, opened a new log
I20260620 18:34:14.107690 10765 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } }
I20260620 18:34:14.107853 10765 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:14.107887 10765 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1cd5a4afc7945db9962d3427295eae7, State: Initialized, Role: FOLLOWER
I20260620 18:34:14.107995 10765 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } }
I20260620 18:34:14.108075 10765 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:14.108107 10765 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:14.108152 10765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:14.108763 10765 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } }
I20260620 18:34:14.108861 10765 leader_election.cc:302] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1cd5a4afc7945db9962d3427295eae7; no voters:
I20260620 18:34:14.108999 10765 leader_election.cc:288] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:14.109031 10770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:14.109251 10770 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [term 1 LEADER]: Becoming Leader. State: Replica: b1cd5a4afc7945db9962d3427295eae7, State: Running, Role: LEADER
I20260620 18:34:14.109258 10765 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:14.110386 10770 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } }
I20260620 18:34:14.110988 10772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1cd5a4afc7945db9962d3427295eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } } }
I20260620 18:34:14.111131 10772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:14.111243 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1cd5a4afc7945db9962d3427295eae7. Latest consensus state: current_term: 1 leader_uuid: "b1cd5a4afc7945db9962d3427295eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd5a4afc7945db9962d3427295eae7" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42289 } } }
I20260620 18:34:14.111308 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7 [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:14.111487 10784 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:14.111545 10784 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:14.111858 10786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:14.112073 10786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:14.113498 10786 catalog_manager.cc:1347] Generated new cluster ID: c6b7297d40044f86897eb6a8de30403a
I20260620 18:34:14.113548 10786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:14.118116 10786 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:14.118691 10786 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:14.119726 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.103129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.10.109.126:42289)
0620 18:34:14.103287 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.103297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.103538 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:34:14.105759 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.105766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.105820 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.105965 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.107698 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.109027 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.110321 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.110942 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.111039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.115375 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.115379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.115379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.115390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.117339 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.117968 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.117970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.117971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.118098 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.118396 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.118398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.118398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.118758 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.118908 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.119340 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.119489 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260620 18:34:14.123637 10786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1cd5a4afc7945db9962d3427295eae7: Generated new TSK 0
I20260620 18:34:14.123795 10786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:14.153826 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:14.235056 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:14.235247 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:14.235271 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:14.236332 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:14.236363 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:14.236389 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:14.237713 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:14.237885 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:14.238049 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:14.243078 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.243129 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.243307 10791 server_base.cc:1034] running on GCE node
W20260620 18:34:14.243129 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.243547 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:14.243789 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:14.244935 10791 hybrid_clock.cc:648] HybridClock initialized: now 1781980454244923 us; error 29 us; skew 500 ppm
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:14.247390 10791 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:14.247747 10791 webserver.cc:460] Webserver started at http://127.10.109.65:35275/ using document root <none> and password file <none>
I20260620 18:34:14.247979 10791 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:14.248046 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:14.248194 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:14.249176 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "725922096d4a482187128f22966e0a69"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "47b8d831aff0873ef4c82553b7027288"
server_key_iv: "44ee9a278ed3c8b26e650838954e8929"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.249513 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "725922096d4a482187128f22966e0a69"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "47b8d831aff0873ef4c82553b7027288"
server_key_iv: "44ee9a278ed3c8b26e650838954e8929"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.250864 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:14.251572 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.251744 10791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:14.251842 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "725922096d4a482187128f22966e0a69"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "47b8d831aff0873ef4c82553b7027288"
server_key_iv: "44ee9a278ed3c8b26e650838954e8929"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.251924 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:14.273828 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:14.276008 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:14.276166 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:14.276378 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:14.276762 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:14.276816 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.276855 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:14.276885 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.282384 10791 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:39543
I20260620 18:34:14.282461 10919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:39543 every 8 connection(s)
I20260620 18:34:14.282775 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:14.288930 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 10791
I20260620 18:34:14.289027 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:14.289201 10677 external_mini_cluster.cc:1428] Setting key 6d92f21b85daad14dee20f799d2858a2
I20260620 18:34:14.291970 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.283496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42289 (local address 127.10.109.65:54303)
0620 18:34:14.283667 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:14.283699 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:14.284122 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:14.284308 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:14.284311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:14.284825 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:14.284999 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.285003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.285664 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.285665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.286409 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.286411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.286848 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.286849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.286951 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.287427 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:14.287435 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:14.288713 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:14.290435 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.290441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.290443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.290787 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.290976 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.290977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.290978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.291113 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.291491 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:14.291497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:14.291606 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 18:34:14.291700 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:34:14.291813 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":114,"threads_started":1}
I20260620 18:34:14.291973 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.283527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:54303 (local address 127.10.109.126:42289)
0620 18:34:14.283594 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.283596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.283778 (+ 182us) server_negotiation.cc:407] Connection header received
0620 18:34:14.284203 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.284204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.284214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.284262 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.285087 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.285580 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.286484 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.286709 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.286845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.288805 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.288806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.288806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.288812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.290353 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.290858 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.290859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.290859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.290911 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.291188 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.291189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.291189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.291357 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.291434 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.291765 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.291855 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:34:14.292477 10920 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42289
I20260620 18:34:14.292579 10920 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:14.292754 10920 heartbeater.cc:511] Master 127.10.109.126:42289 requested a full tablet report, sending...
I20260620 18:34:14.293197 10729 ts_manager.cc:194] Registered new tserver with Master: 725922096d4a482187128f22966e0a69 (127.10.109.65:39543)
I20260620 18:34:14.293892 10729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:54303
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:14.314216 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:14.393688 10925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:14.393883 10925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:14.393906 10925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:14.394991 10925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:14.395025 10925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:14.395051 10925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:14.396390 10925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:14.396562 10925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:14.396731 10925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:14.401465 10931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.401471 10933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.401664 10930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.401803 10925 server_base.cc:1034] running on GCE node
I20260620 18:34:14.402061 10925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:14.402254 10925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:14.403399 10925 hybrid_clock.cc:648] HybridClock initialized: now 1781980454403378 us; error 38 us; skew 500 ppm
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:14.405857 10925 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:14.406318 10925 webserver.cc:460] Webserver started at http://127.10.109.66:39431/ using document root <none> and password file <none>
I20260620 18:34:14.406503 10925 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:14.406543 10925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:14.406635 10925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:14.407676 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "09524415069649e28ae6a975a9f1e0a5"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "0d73fa4eb6ddfb02647f040c1b7dd148"
server_key_iv: "d6a31293f7bd21edff748a7254082685"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.408017 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "09524415069649e28ae6a975a9f1e0a5"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "0d73fa4eb6ddfb02647f040c1b7dd148"
server_key_iv: "d6a31293f7bd21edff748a7254082685"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.409426 10925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:14.410316 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.410542 10925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:14.410655 10925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "09524415069649e28ae6a975a9f1e0a5"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "0d73fa4eb6ddfb02647f040c1b7dd148"
server_key_iv: "d6a31293f7bd21edff748a7254082685"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.410734 10925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:14.440048 10925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:14.442488 10925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:14.442665 10925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:14.442945 10925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:14.443324 10925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:14.443384 10925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.443442 10925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:14.443475 10925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.449307 10925 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:40995
I20260620 18:34:14.449340 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:40995 every 8 connection(s)
I20260620 18:34:14.449657 10925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:14.458433 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.450373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:59335 (local address 127.10.109.126:42289)
0620 18:34:14.450439 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.450441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.450606 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:34:14.451134 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.451135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.451147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.451210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.452107 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.452577 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.453538 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.453728 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.453782 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.455917 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.455918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.455919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.455925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.457324 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.457674 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.457675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.457675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.457724 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.457930 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.457931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.457931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.458024 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.458072 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.458315 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.458372 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:34:14.458635 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.450331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42289 (local address 127.10.109.66:59335)
0620 18:34:14.450476 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:14.450517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:14.451051 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:14.451264 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:14.451267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:14.451794 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:14.452033 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.452037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.452656 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.452657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.453464 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.453466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.453831 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.453832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.453973 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.454411 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:14.454419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:14.455838 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:14.457399 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.457404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.457406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.457607 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.457782 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.457783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.457783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.457875 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.458089 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:14.458092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:14.458208 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 18:34:14.458290 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:34:14.458398 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260620 18:34:14.458725 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 10925
I20260620 18:34:14.458820 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:14.459004 10677 external_mini_cluster.cc:1428] Setting key 2759d0649cf7d1284e552e263157fb62
I20260620 18:34:14.459194 11054 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42289
I20260620 18:34:14.459303 11054 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:14.459465 11054 heartbeater.cc:511] Master 127.10.109.126:42289 requested a full tablet report, sending...
I20260620 18:34:14.460951 10729 ts_manager.cc:194] Registered new tserver with Master: 09524415069649e28ae6a975a9f1e0a5 (127.10.109.66:40995)
I20260620 18:34:14.461325 10729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:59335
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:14.481364 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:14.561087 11059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:14.561280 11059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:14.561301 11059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:14.562441 11059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:14.562481 11059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:14.562510 11059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:14.563980 11059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:14.564224 11059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:14.564457 11059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:14.569332 11065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.569407 11059 server_base.cc:1034] running on GCE node
W20260620 18:34:14.569334 11067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:14.569351 11064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:14.569842 11059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:14.570031 11059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:14.571162 11059 hybrid_clock.cc:648] HybridClock initialized: now 1781980454571147 us; error 27 us; skew 500 ppm
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:14.573638 11059 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:14.574059 11059 webserver.cc:460] Webserver started at http://127.10.109.67:43393/ using document root <none> and password file <none>
I20260620 18:34:14.574291 11059 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:14.574335 11059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:14.574496 11059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:14.575570 11059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "65add28268854a99af21a03954aed6dc"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "db57d42a78ae4ff2f28abcdb699518f7"
server_key_iv: "d8c20a10bf55b63c743074015c5d1256"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.575920 11059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "65add28268854a99af21a03954aed6dc"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "db57d42a78ae4ff2f28abcdb699518f7"
server_key_iv: "d8c20a10bf55b63c743074015c5d1256"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.577358 11059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:14.578164 11074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.578431 11059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:14.578526 11059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "65add28268854a99af21a03954aed6dc"
format_stamp: "Formatted at 2026-06-20 18:34:14 on dist-test-slave-3jql"
server_key: "db57d42a78ae4ff2f28abcdb699518f7"
server_key_iv: "d8c20a10bf55b63c743074015c5d1256"
server_key_version: "encryptionkey@0"
I20260620 18:34:14.578601 11059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:14.589708 11059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:14.591852 11059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:14.592006 11059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:14.592213 11059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:14.592502 11059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:14.592556 11059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.592587 11059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:14.592614 11059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:14.598348 11059 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40115
I20260620 18:34:14.598385 11187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40115 every 8 connection(s)
I20260620 18:34:14.598685 11059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:14.605788 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11059
I20260620 18:34:14.605870 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:14.606058 10677 external_mini_cluster.cc:1428] Setting key f17dfe00528465d8d8a096f143bf32dd
I20260620 18:34:14.607820 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.599396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:60603 (local address 127.10.109.126:42289)
0620 18:34:14.599445 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.599448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.599662 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:34:14.600138 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.600139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.600149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.600199 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.601126 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.601609 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.602584 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.602804 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.602864 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.604850 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.604851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.604851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.604858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.606434 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.606954 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.606955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.606956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.607013 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.607312 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.607313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.607313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.607423 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.607480 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.607701 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.607758 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:14.608028 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.599389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42289 (local address 127.10.109.67:60603)
0620 18:34:14.599546 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:14.599578 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:14.600028 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:14.600247 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:14.600250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:14.600797 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:14.601024 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.601030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.601688 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.601688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.602477 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:14.602479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.602882 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.602883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:14.602992 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.603423 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:14.603431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:14.604765 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:14.606521 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.606528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.606530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.606862 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.607087 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:14.607088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:14.607089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:14.607235 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:14.607482 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:14.607485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:14.607587 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 18:34:14.607681 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:34:14.607792 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260620 18:34:14.608467 11188 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42289
I20260620 18:34:14.608539 11188 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:14.608731 11188 heartbeater.cc:511] Master 127.10.109.126:42289 requested a full tablet report, sending...
I20260620 18:34:14.609091 10729 ts_manager.cc:194] Registered new tserver with Master: 65add28268854a99af21a03954aed6dc (127.10.109.67:40115)
I20260620 18:34:14.609491 10729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:60603
I20260620 18:34:14.618077 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:14.640681 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.632971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.10.109.65:39543)
0620 18:34:14.633145 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.633150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.633160 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:14.633193 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.633195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.633209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.633313 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.633873 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.634594 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.635397 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.635720 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.635810 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.637971 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.637979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.637980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.637987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.639433 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.639822 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.639825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.639826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.639884 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.640087 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.640088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.640088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.640255 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.640438 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.640480 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.640557 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
W20260620 18:34:14.641026 10834 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34078
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[10684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:14.664867 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:14.656767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.10.109.65:39543)
0620 18:34:14.656850 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:14.656852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:14.656862 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:14.657114 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:14.657123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:14.657136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:14.657190 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:14.657813 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.658349 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.659299 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:14.659492 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:14.659552 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:14.662275 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:14.662278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:14.662279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:14.662294 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:14.663853 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.664214 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.664215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.664215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.664274 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:14.664445 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:14.664445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:14.664445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:14.664539 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:14.664626 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:14.664704 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:34:14.664765 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:34:14.665580 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 10791
I20260620 18:34:14.670584 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 10925
I20260620 18:34:14.676090 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11059
I20260620 18:34:14.680202 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 10697
2026-06-20T18:34:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1887 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[11202](info): set up 2 sockets
Jun 20 18:34:14 dist-test-slave-3jql krb5kdc[11202](info): commencing operation
krb5kdc: starting...
W20260620 18:34:15.718807 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:15 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:15.835819 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40389
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:15.916204 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:15.916455 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:15.916500 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:15.917635 11215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:15.917698 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:15.917723 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:15.917744 11215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:15.917771 11215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:15.919242 11215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:15.919512 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:15.919730 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:15.925760 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.926056 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.926149 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:15.926366 11215 server_base.cc:1034] running on GCE node
I20260620 18:34:15.926637 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:15.926990 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:15.928205 11215 hybrid_clock.cc:648] HybridClock initialized: now 1781980455928176 us; error 40 us; skew 500 ppm
Jun 20 18:34:15 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:15.930889 11215 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:15.931308 11215 webserver.cc:460] Webserver started at http://127.10.109.126:44849/ using document root <none> and password file <none>
I20260620 18:34:15.931511 11215 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:15.931574 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:15.931730 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:15.932772 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "871d3ab3a0274e62a752c48dbacb879d"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-3jql"
server_key: "009f226ed84349748e28d3ab977a4f02"
server_key_iv: "a620acc362e0cbfa21e577a0c8baba94"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.933126 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "871d3ab3a0274e62a752c48dbacb879d"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-3jql"
server_key: "009f226ed84349748e28d3ab977a4f02"
server_key_iv: "a620acc362e0cbfa21e577a0c8baba94"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.934504 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:15.935508 11230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.935985 11215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:15.936107 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "871d3ab3a0274e62a752c48dbacb879d"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-3jql"
server_key: "009f226ed84349748e28d3ab977a4f02"
server_key_iv: "a620acc362e0cbfa21e577a0c8baba94"
server_key_version: "encryptionkey@0"
I20260620 18:34:15.936406 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:15.948313 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:15.950511 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:15.950627 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:15.954339 11215 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:40389
I20260620 18:34:15.954399 11282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:40389 every 8 connection(s)
I20260620 18:34:15.954782 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:15.955651 11283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:15.957888 11283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d: Bootstrap starting.
I20260620 18:34:15.958462 11283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:15.958819 11283 log.cc:824] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:15.959518 11283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d: No bootstrap required, opened a new log
I20260620 18:34:15.960201 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11215
I20260620 18:34:15.960301 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:15.960477 10677 external_mini_cluster.cc:1428] Setting key 2ab50844f269635ea402f981bd506528
I20260620 18:34:15.961426 11283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } }
I20260620 18:34:15.961575 11283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:15.961612 11283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871d3ab3a0274e62a752c48dbacb879d, State: Initialized, Role: FOLLOWER
I20260620 18:34:15.961741 11283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } }
I20260620 18:34:15.961826 11283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:15.961865 11283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:15.961918 11283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:15.962791 11283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } }
I20260620 18:34:15.962914 11283 leader_election.cc:302] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871d3ab3a0274e62a752c48dbacb879d; no voters:
I20260620 18:34:15.963128 11288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:15.963114 11283 leader_election.cc:288] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:15.963275 11288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [term 1 LEADER]: Becoming Leader. State: Replica: 871d3ab3a0274e62a752c48dbacb879d, State: Running, Role: LEADER
I20260620 18:34:15.963359 11288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } }
I20260620 18:34:15.963420 11283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:15.963678 11289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871d3ab3a0274e62a752c48dbacb879d. Latest consensus state: current_term: 1 leader_uuid: "871d3ab3a0274e62a752c48dbacb879d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } } }
I20260620 18:34:15.963759 11289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:15.963999 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871d3ab3a0274e62a752c48dbacb879d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d3ab3a0274e62a752c48dbacb879d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40389 } } }
I20260620 18:34:15.964109 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:15.964290 11293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:15.964768 11293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:15.966401 11293 catalog_manager.cc:1347] Generated new cluster ID: 177db001a1cf4e14a994bda805a0cf23
I20260620 18:34:15.966452 11293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:15 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:15.972074 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:15.961010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.10.109.126:40389)
0620 18:34:15.961194 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:15.961209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:15.961232 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:34:15.961390 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:15.961395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:15.961440 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:15.961570 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:15.962267 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.963407 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.964465 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.965165 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.965271 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:15.967962 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:15.967966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:15.967967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:15.967978 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:15.970078 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.970469 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.970472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.970472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.970615 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.970823 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.970823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.970824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.971124 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:15.971673 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:15.971743 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:34:15.971878 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260620 18:34:15.989759 11293 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:15.990455 11293 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:15.993799 11293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871d3ab3a0274e62a752c48dbacb879d: Generated new TSK 0
I20260620 18:34:15.993953 11293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:16.022576 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:16.101883 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:16.102072 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:16.102094 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:16.103174 11309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:16.103206 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:16.103231 11309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:16.104590 11309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:16.104751 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:16.104915 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:16.109838 11315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.109884 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.109849 11317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.110234 11309 server_base.cc:1034] running on GCE node
I20260620 18:34:16.110416 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:16.110611 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:16.111773 11309 hybrid_clock.cc:648] HybridClock initialized: now 1781980456111762 us; error 42 us; skew 500 ppm
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:16.114179 11309 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:16.114601 11309 webserver.cc:460] Webserver started at http://127.10.109.65:41821/ using document root <none> and password file <none>
I20260620 18:34:16.114866 11309 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:16.114933 11309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:16.115093 11309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:16.116115 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "6e0153c0b0354b07aa20ebe490cd14a6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "99666628870fdb82c78ed1acfcd061f2"
server_key_iv: "3a2948eed9c51e235d3c71e30299b61a"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.116467 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "6e0153c0b0354b07aa20ebe490cd14a6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "99666628870fdb82c78ed1acfcd061f2"
server_key_iv: "3a2948eed9c51e235d3c71e30299b61a"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.117791 11309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:16.118549 11324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.118793 11309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:16.118904 11309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "6e0153c0b0354b07aa20ebe490cd14a6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "99666628870fdb82c78ed1acfcd061f2"
server_key_iv: "3a2948eed9c51e235d3c71e30299b61a"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.118992 11309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:16.148680 11309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:16.150770 11309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:16.150931 11309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:16.151187 11309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:16.151496 11309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:16.151551 11309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.151593 11309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:16.151618 11309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.157284 11309 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:42815
I20260620 18:34:16.157325 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:42815 every 8 connection(s)
I20260620 18:34:16.157646 11309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:16.166873 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.158395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:47611 (local address 127.10.109.126:40389)
0620 18:34:16.158458 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.158460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.158696 (+ 236us) server_negotiation.cc:407] Connection header received
0620 18:34:16.159671 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.159672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.159688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.159739 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.160327 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.160818 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.161818 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.162027 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.162133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.164216 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.164218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.164218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.164225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.165669 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.166016 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.166017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.166017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.166064 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.166281 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.166282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.166282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.166376 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.166425 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.166636 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.166785 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:34:16.166958 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.158354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40389 (local address 127.10.109.65:47611)
0620 18:34:16.158541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.158589 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.159543 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.159806 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.159810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.160048 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:16.160223 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.160227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.160908 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.160909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.161693 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.161695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.162135 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.162136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.162305 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.162716 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:16.162723 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:16.164136 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:16.165742 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.165748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.165750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.165952 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.166120 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.166121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.166121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.166227 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.166429 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:16.166433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:16.166533 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.166608 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.166732 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260620 18:34:16.167450 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11309
I20260620 18:34:16.167485 11438 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40389
I20260620 18:34:16.167541 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:16.167583 11438 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:16.167744 10677 external_mini_cluster.cc:1428] Setting key b34c4c02ad25f1a8eda4fb86d6fa4bd8
I20260620 18:34:16.167755 11438 heartbeater.cc:511] Master 127.10.109.126:40389 requested a full tablet report, sending...
I20260620 18:34:16.168254 11247 ts_manager.cc:194] Registered new tserver with Master: 6e0153c0b0354b07aa20ebe490cd14a6 (127.10.109.65:42815)
I20260620 18:34:16.168831 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:47611
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:16.191697 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:16.270668 11443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:16.270864 11443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:16.270886 11443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:16.271950 11443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:16.271983 11443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:16.272009 11443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:16.273360 11443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:16.273525 11443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:16.273691 11443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:16.278712 11448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.278852 11443 server_base.cc:1034] running on GCE node
W20260620 18:34:16.278803 11449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.278712 11451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.279222 11443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:16.279455 11443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:16.280629 11443 hybrid_clock.cc:648] HybridClock initialized: now 1781980456280609 us; error 37 us; skew 500 ppm
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:16.283026 11443 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:16.283423 11443 webserver.cc:460] Webserver started at http://127.10.109.66:36729/ using document root <none> and password file <none>
I20260620 18:34:16.283651 11443 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:16.283718 11443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:16.283864 11443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:16.284842 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "8389b75953de4460b245d95aec70914e"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "a73840fdc562bfa99fa482a40efe7f8b"
server_key_iv: "af591abc86c231b20c0a0977ba918d5f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.285187 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "8389b75953de4460b245d95aec70914e"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "a73840fdc562bfa99fa482a40efe7f8b"
server_key_iv: "af591abc86c231b20c0a0977ba918d5f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.286461 11443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:16.287281 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.287465 11443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.287562 11443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "8389b75953de4460b245d95aec70914e"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "a73840fdc562bfa99fa482a40efe7f8b"
server_key_iv: "af591abc86c231b20c0a0977ba918d5f"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.287655 11443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:16.317360 11443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:16.319708 11443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:16.319826 11443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:16.320006 11443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:16.320343 11443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:16.320384 11443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.320407 11443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:16.320420 11443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.326148 11443 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:42951
I20260620 18:34:16.326247 11571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:42951 every 8 connection(s)
I20260620 18:34:16.326565 11443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:16.336105 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.327333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:37479 (local address 127.10.109.126:40389)
0620 18:34:16.327384 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.327387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.327604 (+ 217us) server_negotiation.cc:407] Connection header received
0620 18:34:16.328519 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.328520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.328530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.328608 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.329199 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.329920 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.331255 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.331433 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.331491 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.333570 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.333571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.333571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.333577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.335002 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.335369 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.335370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.335370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.335420 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.335618 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.335618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.335619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.335716 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.335765 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.335980 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.336042 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:16.336293 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.327323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40389 (local address 127.10.109.66:37479)
0620 18:34:16.327486 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.327521 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.328424 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.328689 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.328692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.328933 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:16.329117 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.329120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.330043 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.330044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.331183 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.331185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.331558 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.331559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.331669 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.332192 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:16.332200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:16.333483 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:16.335068 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.335073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.335075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.335299 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.335478 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.335478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.335478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.335567 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.335769 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:16.335773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:16.335876 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.335955 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.336063 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260620 18:34:16.336481 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11443
I20260620 18:34:16.336575 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:16.336761 10677 external_mini_cluster.cc:1428] Setting key 8d126ad7ef489583b58ea88e24d455a1
I20260620 18:34:16.336781 11572 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40389
I20260620 18:34:16.336867 11572 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:16.337038 11572 heartbeater.cc:511] Master 127.10.109.126:40389 requested a full tablet report, sending...
I20260620 18:34:16.337387 11247 ts_manager.cc:194] Registered new tserver with Master: 8389b75953de4460b245d95aec70914e (127.10.109.66:42951)
I20260620 18:34:16.337764 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:37479
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:16.361191 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:16.439942 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:16.440140 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:16.440162 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:16.441293 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:16.441332 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:16.441358 11577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:16.442924 11577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:16.443150 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:16.443329 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:16.448220 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.448220 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.448467 11577 server_base.cc:1034] running on GCE node
W20260620 18:34:16.448263 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.448769 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:16.449002 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:16.450225 11577 hybrid_clock.cc:648] HybridClock initialized: now 1781980456450214 us; error 29 us; skew 500 ppm
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:16.452798 11577 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:16.453217 11577 webserver.cc:460] Webserver started at http://127.10.109.67:34841/ using document root <none> and password file <none>
I20260620 18:34:16.453450 11577 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:16.453518 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:16.453670 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:16.454726 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "d2005d7055344ad587179065497602f6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "7f200774e66a920faa4cd76c777a2dcf"
server_key_iv: "2dde0c26a09901dcc2b4c7f9fa2877ff"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.455145 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "d2005d7055344ad587179065497602f6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "7f200774e66a920faa4cd76c777a2dcf"
server_key_iv: "2dde0c26a09901dcc2b4c7f9fa2877ff"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.456449 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:16.457247 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.457430 11577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:16.457535 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "d2005d7055344ad587179065497602f6"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-3jql"
server_key: "7f200774e66a920faa4cd76c777a2dcf"
server_key_iv: "2dde0c26a09901dcc2b4c7f9fa2877ff"
server_key_version: "encryptionkey@0"
I20260620 18:34:16.457617 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:16.464378 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:16.466133 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:16.466290 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:16.466483 11577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:16.466856 11577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:16.466908 11577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.466948 11577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:16.466975 11577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:16.472535 11577 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:41947
I20260620 18:34:16.472597 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:41947 every 8 connection(s)
I20260620 18:34:16.472906 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:16.475368 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11577
I20260620 18:34:16.475442 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:16.475587 10677 external_mini_cluster.cc:1428] Setting key 550a2d5ecc40b8258066fd465d5007e5
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:16.482295 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.473632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:47355 (local address 127.10.109.126:40389)
0620 18:34:16.473732 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.473734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.473903 (+ 169us) server_negotiation.cc:407] Connection header received
0620 18:34:16.474909 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.474910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.474923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.474985 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.475629 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.476503 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.477460 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.477670 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.477742 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.479747 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.479749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.479749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.479757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.480963 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.481351 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.481352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.481352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.481409 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.481665 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.481666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.481667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.481779 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.481856 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.482115 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.482200 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:34:16.482492 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.473614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40389 (local address 127.10.109.67:47355)
0620 18:34:16.473783 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.473825 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.474796 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.475051 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.475055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.475304 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:16.475540 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.475545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.476579 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.476580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.477359 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.477361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.477747 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.477748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.477876 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.478303 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:16.478311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:16.479654 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:16.481048 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.481054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.481056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.481272 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.481483 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:16.481484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:16.481485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:16.481584 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:16.481853 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:16.481859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:16.481987 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.482092 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.482277 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260620 18:34:16.482939 11706 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40389
I20260620 18:34:16.483031 11706 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:16.483198 11706 heartbeater.cc:511] Master 127.10.109.126:40389 requested a full tablet report, sending...
I20260620 18:34:16.483546 11247 ts_manager.cc:194] Registered new tserver with Master: d2005d7055344ad587179065497602f6 (127.10.109.67:41947)
I20260620 18:34:16.483925 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:47355
I20260620 18:34:16.487435 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:16.495617 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.488918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.10.109.126:40389)
0620 18:34:16.488968 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.488969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.489086 (+ 117us) server_negotiation.cc:407] Connection header received
0620 18:34:16.489158 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.489159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.489167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.489213 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.489697 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.490121 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.490963 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.491156 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.491212 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.492407 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.492408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.492408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.492414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.493515 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.494172 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.494174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.494174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.494273 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.494561 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.494562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.494563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.494987 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.495089 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.495315 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.495550 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:34:16.498028 11247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:16.499675 11247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:16.506186 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.502211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:41947 (local address 127.0.0.1:32868)
0620 18:34:16.502527 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.502534 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.502563 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.503069 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.503070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.503072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.503121 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.503123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.503954 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.503955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.505114 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.505116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.505914 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.505915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.505977 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.505987 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.506061 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.506134 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":95,"thread_start_us":68,"threads_started":1}
I20260620 18:34:16.506217 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.502309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.10.109.67:41947)
0620 18:34:16.502828 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.502833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.502841 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:16.502877 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.502879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.502893 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.502979 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.503199 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.503854 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.505185 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.505814 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.505893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.505946 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.506099 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.506140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":99,"threads_started":1}
I20260620 18:34:16.506855 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.501984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42815 (local address 127.0.0.1:54696)
0620 18:34:16.502314 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.502352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.502416 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.502685 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.502688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.502690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.502786 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.502789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.504065 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.504066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.505368 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.505371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.506575 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.506576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.506654 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.506669 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.506731 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.506789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":64,"thread_start_us":80,"threads_started":1}
I20260620 18:34:16.506896 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.502132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.10.109.65:42815)
0620 18:34:16.502491 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.502495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.502507 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:34:16.502550 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.502553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.502568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.502681 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.502873 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.503960 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.505456 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.506193 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.506267 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.506326 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.506777 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.506820 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":86,"threads_started":1}
I20260620 18:34:16.507128 11640 tablet_service.cc:1467] Processing CreateTablet for tablet 3423cd27679a423fa0418da6a5b549c5 (DEFAULT_TABLE table=test-table [id=68fe0e7e4b704db0bcb21af2f5d5a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.507481 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3423cd27679a423fa0418da6a5b549c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.507504 11372 tablet_service.cc:1467] Processing CreateTablet for tablet 3423cd27679a423fa0418da6a5b549c5 (DEFAULT_TABLE table=test-table [id=68fe0e7e4b704db0bcb21af2f5d5a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.507779 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.502376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:42951 (local address 127.0.0.1:53056)
0620 18:34:16.502648 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.502656 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.502691 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.502988 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.502989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.502991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.503030 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.503032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.504228 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.504229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.505368 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.505371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.507524 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.507525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.507582 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.507585 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.507670 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.507712 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260620 18:34:16.507805 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.502493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.10.109.66:42951)
0620 18:34:16.502676 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.502680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.502691 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:16.502779 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.502780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.502797 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.502889 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.503111 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.504142 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.506363 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.507362 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.507425 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.507485 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.507688 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.507727 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260620 18:34:16.507822 11372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3423cd27679a423fa0418da6a5b549c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.510911 11723 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: Bootstrap starting.
I20260620 18:34:16.511564 11723 tablet_bootstrap.cc:654] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.511528 11506 tablet_service.cc:1467] Processing CreateTablet for tablet 3423cd27679a423fa0418da6a5b549c5 (DEFAULT_TABLE table=test-table [id=68fe0e7e4b704db0bcb21af2f5d5a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:16.511837 11506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3423cd27679a423fa0418da6a5b549c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:16.512084 11723 log.cc:824] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.512379 11724 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: Bootstrap starting.
I20260620 18:34:16.512938 11723 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: No bootstrap required, opened a new log
I20260620 18:34:16.513022 11723 ts_tablet_manager.cc:1397] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:16.513055 11724 tablet_bootstrap.cc:654] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.513782 11724 log.cc:824] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.514612 11724 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: No bootstrap required, opened a new log
I20260620 18:34:16.514683 11724 ts_tablet_manager.cc:1397] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:34:16.514940 11727 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Bootstrap starting.
I20260620 18:34:16.515260 11723 raft_consensus.cc:348] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.515442 11723 raft_consensus.cc:374] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.515482 11723 raft_consensus.cc:729] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0153c0b0354b07aa20ebe490cd14a6, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.515506 11727 tablet_bootstrap.cc:654] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:16.515604 11723 consensus_queue.cc:260] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.515872 11723 ts_tablet_manager.cc:1428] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:16.515905 11438 heartbeater.cc:503] Master 127.10.109.126:40389 was elected leader, sending a full tablet report...
I20260620 18:34:16.516021 11727 log.cc:824] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:16.516803 11727 tablet_bootstrap.cc:492] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: No bootstrap required, opened a new log
I20260620 18:34:16.516878 11727 ts_tablet_manager.cc:1397] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:16.517541 11724 raft_consensus.cc:348] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.517665 11724 raft_consensus.cc:374] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.517685 11724 raft_consensus.cc:729] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2005d7055344ad587179065497602f6, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.517791 11724 consensus_queue.cc:260] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.518031 11724 ts_tablet_manager.cc:1428] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:34:16.518041 11706 heartbeater.cc:503] Master 127.10.109.126:40389 was elected leader, sending a full tablet report...
I20260620 18:34:16.518805 11727 raft_consensus.cc:348] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.518987 11727 raft_consensus.cc:374] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:16.519034 11727 raft_consensus.cc:729] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8389b75953de4460b245d95aec70914e, State: Initialized, Role: FOLLOWER
I20260620 18:34:16.519140 11727 consensus_queue.cc:260] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.519387 11727 ts_tablet_manager.cc:1428] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:34:16.519443 11572 heartbeater.cc:503] Master 127.10.109.126:40389 was elected leader, sending a full tablet report...
I20260620 18:34:16.547608 11731 raft_consensus.cc:482] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:16.547719 11731 raft_consensus.cc:504] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.548048 11731 leader_election.cc:288] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e0153c0b0354b07aa20ebe490cd14a6 (127.10.109.65:42815), d2005d7055344ad587179065497602f6 (127.10.109.67:41947)
I20260620 18:34:16.552112 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.548560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:53035 (local address 127.10.109.67:41947)
0620 18:34:16.548598 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.548600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.548696 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:34:16.548737 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.548737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.548746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.548796 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.548918 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.549608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551117 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.551699 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551778 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.551787 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.552001 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.552042 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:34:16.552112 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.548487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:43285 (local address 127.10.109.65:42815)
0620 18:34:16.548540 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.548542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.548551 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:16.548578 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.548579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.548590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.548663 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:16.548815 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.549600 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551117 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.551692 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551779 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.551787 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.552001 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.552042 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:34:16.552138 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.548481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:41947 (local address 127.10.109.66:53035)
0620 18:34:16.548612 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.548626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.548686 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.548795 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.548796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.548798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.548850 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.548851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.549695 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.549696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.551019 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.551022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.551817 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.551907 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.551912 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.551985 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.552032 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":58,"thread_start_us":89,"threads_started":1}
I20260620 18:34:16.552347 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.548363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42815 (local address 127.10.109.66:43285)
0620 18:34:16.548421 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:16.548432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:16.548475 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:16.548672 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:16.548673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:16.548675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:16.548749 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.548750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.549695 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.549696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.551019 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:16.551022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.551817 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.551819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:16.551912 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.551914 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:16.551985 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:34:16.552034 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":38}
I20260620 18:34:16.552510 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3423cd27679a423fa0418da6a5b549c5" candidate_uuid: "8389b75953de4460b245d95aec70914e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2005d7055344ad587179065497602f6" is_pre_election: true
I20260620 18:34:16.552677 11660 raft_consensus.cc:2454] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8389b75953de4460b245d95aec70914e in term 0.
I20260620 18:34:16.552922 11460 leader_election.cc:302] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8389b75953de4460b245d95aec70914e, d2005d7055344ad587179065497602f6; no voters:
I20260620 18:34:16.552951 11392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3423cd27679a423fa0418da6a5b549c5" candidate_uuid: "8389b75953de4460b245d95aec70914e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" is_pre_election: true
I20260620 18:34:16.553067 11731 raft_consensus.cc:2781] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:16.553114 11731 raft_consensus.cc:482] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:16.553131 11392 raft_consensus.cc:2454] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8389b75953de4460b245d95aec70914e in term 0.
I20260620 18:34:16.553153 11731 raft_consensus.cc:3037] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.553857 11731 raft_consensus.cc:504] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.554003 11731 leader_election.cc:288] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [CANDIDATE]: Term 1 election: Requested vote from peers 6e0153c0b0354b07aa20ebe490cd14a6 (127.10.109.65:42815), d2005d7055344ad587179065497602f6 (127.10.109.67:41947)
I20260620 18:34:16.554164 11392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3423cd27679a423fa0418da6a5b549c5" candidate_uuid: "8389b75953de4460b245d95aec70914e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0153c0b0354b07aa20ebe490cd14a6"
I20260620 18:34:16.554211 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3423cd27679a423fa0418da6a5b549c5" candidate_uuid: "8389b75953de4460b245d95aec70914e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2005d7055344ad587179065497602f6"
I20260620 18:34:16.554266 11392 raft_consensus.cc:3037] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.554261 11660 raft_consensus.cc:3037] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:16.554988 11660 raft_consensus.cc:2454] T 3423cd27679a423fa0418da6a5b549c5 P d2005d7055344ad587179065497602f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8389b75953de4460b245d95aec70914e in term 1.
I20260620 18:34:16.555078 11392 raft_consensus.cc:2454] T 3423cd27679a423fa0418da6a5b549c5 P 6e0153c0b0354b07aa20ebe490cd14a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8389b75953de4460b245d95aec70914e in term 1.
I20260620 18:34:16.555193 11460 leader_election.cc:302] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8389b75953de4460b245d95aec70914e, d2005d7055344ad587179065497602f6; no voters:
I20260620 18:34:16.555282 11731 raft_consensus.cc:2781] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:16.555444 11731 raft_consensus.cc:686] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [term 1 LEADER]: Becoming Leader. State: Replica: 8389b75953de4460b245d95aec70914e, State: Running, Role: LEADER
I20260620 18:34:16.555557 11731 consensus_queue.cc:237] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } }
I20260620 18:34:16.556409 11247 catalog_manager.cc:5495] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8389b75953de4460b245d95aec70914e (127.10.109.66). New cstate: current_term: 1 leader_uuid: "8389b75953de4460b245d95aec70914e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0153c0b0354b07aa20ebe490cd14a6" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2005d7055344ad587179065497602f6" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8389b75953de4460b245d95aec70914e" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 42951 } health_report { overall_health: HEALTHY } } }
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:16.569597 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.560464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.10.109.65:42815)
0620 18:34:16.560509 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.560511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.560525 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:34:16.560587 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.560588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.560595 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.560647 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.561269 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.561710 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.562386 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.562560 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.562629 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.565039 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.565046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.565046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.565052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.566987 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.567643 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.567646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.567646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.567702 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.567989 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.567990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.567990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.568138 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.568257 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.569354 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.569424 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 18:34:16.577176 11573 tablet.cc:2369] T 3423cd27679a423fa0418da6a5b549c5 P 8389b75953de4460b245d95aec70914e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:16.593847 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.586345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.10.109.65:42815)
0620 18:34:16.586466 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.586468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.586525 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:34:16.586570 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.586571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.586580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.586648 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.587250 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.587807 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.588596 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.588838 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.588900 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.591239 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.591240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.591241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.591247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.592436 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.592847 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.592849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.592850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.592974 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.593248 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.593249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.593250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.593363 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.593463 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.593665 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.593730 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:16.616168 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:16.609105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.10.109.65:42815)
0620 18:34:16.609259 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:16.609261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:16.609279 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:34:16.609311 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:16.609313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:16.609325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:16.609421 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:16.609992 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.610474 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.611256 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:16.611431 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:16.611490 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:16.613727 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:16.613729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:16.613729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:16.613736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:16.615046 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.615423 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.615424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.615424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.615477 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:16.615688 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:16.615689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:16.615690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:16.615786 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:16.615886 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:16.615991 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 18:34:16.616064 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 18:34:16.616593 11352 tablet_service.cc:2908] Rejecting scan request for tablet 3423cd27679a423fa0418da6a5b549c5: Uninitialized: clean time has not yet been initialized
I20260620 18:34:16.617341 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11309
I20260620 18:34:16.622406 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11443
I20260620 18:34:16.626876 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11577
I20260620 18:34:16.631120 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11215
2026-06-20T18:34:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1951 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11747](info): set up 2 sockets
Jun 20 18:34:16 dist-test-slave-3jql krb5kdc[11747](info): commencing operation
krb5kdc: starting...
W20260620 18:34:17.670885 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:17 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:17.790303 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36109
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:17.871685 11760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:17.871881 11760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:17.871915 11760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:17.873948 11760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:17.874042 11760 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:34:17.874058 11760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:17.874071 11760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:17.874085 11760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:17.874099 11760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:17.876889 11760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:17.877207 11760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:17.877466 11760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:17.882737 11766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:17.882725 11765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:17.882966 11760 server_base.cc:1034] running on GCE node
W20260620 18:34:17.883091 11768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:17.883332 11760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:17.883638 11760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:17.884788 11760 hybrid_clock.cc:648] HybridClock initialized: now 1781980457884766 us; error 39 us; skew 500 ppm
Jun 20 18:34:17 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:17.888059 11760 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:17.888688 11760 webserver.cc:460] Webserver started at http://127.10.109.126:35907/ using document root <none> and password file <none>
I20260620 18:34:17.889025 11760 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:17.889108 11760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:17.889281 11760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:17.890599 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "4f5ecb8b20584d59a593d9d250e3dfee"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-3jql"
server_key: "6262f97b9b3294413f88cd7e4b7fb6bf"
server_key_iv: "3941a4520cdf6c2ae100d0a5f5afb45d"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.891121 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "4f5ecb8b20584d59a593d9d250e3dfee"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-3jql"
server_key: "6262f97b9b3294413f88cd7e4b7fb6bf"
server_key_iv: "3941a4520cdf6c2ae100d0a5f5afb45d"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.892917 11760 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:34:17.893944 11775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.894207 11760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:17.894342 11760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "4f5ecb8b20584d59a593d9d250e3dfee"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-3jql"
server_key: "6262f97b9b3294413f88cd7e4b7fb6bf"
server_key_iv: "3941a4520cdf6c2ae100d0a5f5afb45d"
server_key_version: "encryptionkey@0"
I20260620 18:34:17.894462 11760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:17.912442 11760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:17.914712 11760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:17.914933 11760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:17.919849 11760 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36109
I20260620 18:34:17.919901 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36109 every 8 connection(s)
I20260620 18:34:17.920217 11760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:17.920800 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:17.923115 11838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Bootstrap starting.
I20260620 18:34:17.923722 11838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:17.924010 11838 log.cc:824] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:17.924753 11838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: No bootstrap required, opened a new log
I20260620 18:34:17.925002 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11760
I20260620 18:34:17.925130 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:17.925299 10677 external_mini_cluster.cc:1428] Setting key 4848d351b118be6b15a2e75461559c95
I20260620 18:34:17.927383 11838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } }
I20260620 18:34:17.927517 11838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:17.927541 11838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5ecb8b20584d59a593d9d250e3dfee, State: Initialized, Role: FOLLOWER
I20260620 18:34:17.927714 11838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } }
I20260620 18:34:17.927785 11838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:17.927807 11838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:17.927894 11838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:17.928457 11838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } }
I20260620 18:34:17.928540 11838 leader_election.cc:302] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f5ecb8b20584d59a593d9d250e3dfee; no voters:
I20260620 18:34:17.928771 11838 leader_election.cc:288] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:17.928797 11843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:17.928906 11843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [term 1 LEADER]: Becoming Leader. State: Replica: 4f5ecb8b20584d59a593d9d250e3dfee, State: Running, Role: LEADER
I20260620 18:34:17.929040 11843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } }
I20260620 18:34:17.929034 11838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:17.930203 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f5ecb8b20584d59a593d9d250e3dfee. Latest consensus state: current_term: 1 leader_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } } }
I20260620 18:34:17.930279 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:17.930537 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5ecb8b20584d59a593d9d250e3dfee" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36109 } } }
I20260620 18:34:17.930661 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:17.931038 11859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:17.931080 11859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:17.931141 11855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:17.931290 11855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:34:17 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:17.935598 11855 catalog_manager.cc:1347] Generated new cluster ID: 0bf9c944e694497f88b55951b7adc709
I20260620 18:34:17.935669 11855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:17.937609 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.925826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.10.109.126:36109)
0620 18:34:17.926008 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:17.926023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:17.926048 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:17.927005 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:17.927012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:17.927059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:17.927196 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:17.927902 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.928997 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.930022 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.930677 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.930848 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.933472 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.933476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.933484 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.933495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.935737 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.936148 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.936151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.936152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.936257 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.936465 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.936465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.936466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.936769 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:17.936917 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:17.937259 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:34:17.937400 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260620 18:34:17.947311 11855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:17.947999 11855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:17.954902 11855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Generated new TSK 0
I20260620 18:34:17.955089 11855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:17.983330 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36109
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:18.062889 11864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.063104 11864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.063143 11864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.064299 11864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.064366 11864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.064401 11864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:18.064440 11864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:18.065838 11864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:18.066098 11864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.066321 11864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.071210 11872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.071233 11869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.071208 11870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.071409 11864 server_base.cc:1034] running on GCE node
I20260620 18:34:18.071638 11864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.071875 11864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.073040 11864 hybrid_clock.cc:648] HybridClock initialized: now 1781980458073033 us; error 37 us; skew 500 ppm
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:18.079365 11864 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:18.079902 11864 webserver.cc:460] Webserver started at http://127.10.109.65:46357/ using document root <none> and password file <none>
I20260620 18:34:18.080140 11864 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.080206 11864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.080338 11864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:18.081339 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "8e78b8003128437fa699e8632280f37a"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "82a26a366e8008a7cb58d69bebcc695b"
server_key_iv: "ea9515f5b960f37f374c7f3b1130c1a7"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.081707 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "8e78b8003128437fa699e8632280f37a"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "82a26a366e8008a7cb58d69bebcc695b"
server_key_iv: "ea9515f5b960f37f374c7f3b1130c1a7"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.083106 11864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:34:18.083962 11879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.084161 11864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:18.084266 11864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "8e78b8003128437fa699e8632280f37a"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "82a26a366e8008a7cb58d69bebcc695b"
server_key_iv: "ea9515f5b960f37f374c7f3b1130c1a7"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.084354 11864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.093168 11864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.095940 11864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.096097 11864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.096724 11864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.096788 11864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.096823 11864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.096850 11864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.104899 11864 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:32811
I20260620 18:34:18.104970 11994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:32811 every 8 connection(s)
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:18.105307 11864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:18.108114 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 11864
I20260620 18:34:18.108220 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:18.108397 10677 external_mini_cluster.cc:1428] Setting key a888401c44aa228de172fcb1c1e64371
I20260620 18:34:18.109670 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.097928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:42005 (local address 127.10.109.126:36109)
0620 18:34:18.098005 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.098007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.098020 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:34:18.098357 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.098358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.098387 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.098462 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.099483 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.100524 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.102201 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.102541 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.102633 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.106117 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.106119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.106119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.106128 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.108156 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.108570 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.108572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.108572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.108640 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.108918 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.108919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.108919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.109087 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.109174 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.109464 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.109565 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:34:18.109803 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.097061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.65:42005)
0620 18:34:18.097228 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.097267 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.098178 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.098599 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.098604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.099061 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.099376 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.099382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.100623 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.100624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.102109 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.102112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.102703 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.102704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.102896 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.103648 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.103658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.106040 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.108253 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.108260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.108262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.108478 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.108727 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.108727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.108728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.108847 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.109177 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.109184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.109327 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.109432 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.109586 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260620 18:34:18.110375 11995 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36109
I20260620 18:34:18.110455 11995 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.110637 11995 heartbeater.cc:511] Master 127.10.109.126:36109 requested a full tablet report, sending...
I20260620 18:34:18.111143 11792 ts_manager.cc:194] Registered new tserver with Master: 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811)
I20260620 18:34:18.112048 11792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:42005
I20260620 18:34:18.119395 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.111436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.65:45769)
0620 18:34:18.111923 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.111933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.111993 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.112434 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.112435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.112651 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.112725 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.112727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.113932 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.113933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.114852 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.114855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.115233 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.115234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.115308 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.115769 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.115774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.116171 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.118067 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.118068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.118068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.118300 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.118561 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.118562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.118563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.118646 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.118991 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.118992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.119042 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.119120 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.119238 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":72,"threads_started":1}
I20260620 18:34:18.119395 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.112248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:45769 (local address 127.10.109.126:36109)
0620 18:34:18.112304 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.112306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.112314 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:18.112333 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.112333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.112343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.112394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.113094 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.113799 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.114934 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.115144 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.115232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.116243 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.116245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.116245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.116268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.117978 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.118413 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.118414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.118414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.118472 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.118729 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.118731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.118731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.118890 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.118938 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.119167 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.119258 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:18.141592 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36109
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:18.220507 12004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.220692 12004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.220715 12004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.221855 12004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.221899 12004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.221920 12004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:18.221937 12004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:18.223387 12004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:18.223613 12004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.223847 12004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.228860 12010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.228892 12009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.228910 12012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.229141 12004 server_base.cc:1034] running on GCE node
I20260620 18:34:18.229413 12004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.229671 12004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.230791 12004 hybrid_clock.cc:648] HybridClock initialized: now 1781980458230767 us; error 46 us; skew 500 ppm
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:18.233263 12004 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:18.233705 12004 webserver.cc:460] Webserver started at http://127.10.109.66:42113/ using document root <none> and password file <none>
I20260620 18:34:18.233950 12004 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.234017 12004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.234175 12004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:18.235291 12004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "5f1fc80acd1d4420ac10eb5697874af3"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d91348a6b04e7c945c2a67a769dff859"
server_key_iv: "65f36893232a35dd6d271755ed6c4f9c"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.235673 12004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "5f1fc80acd1d4420ac10eb5697874af3"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d91348a6b04e7c945c2a67a769dff859"
server_key_iv: "65f36893232a35dd6d271755ed6c4f9c"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.237033 12004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:18.237876 12019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.238068 12004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:18.238173 12004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "5f1fc80acd1d4420ac10eb5697874af3"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d91348a6b04e7c945c2a67a769dff859"
server_key_iv: "65f36893232a35dd6d271755ed6c4f9c"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.238257 12004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.252915 12004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.255399 12004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.255554 12004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.256157 12004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.256217 12004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.256251 12004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.256279 12004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:18.263480 12004 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:36453
I20260620 18:34:18.263664 12134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:36453 every 8 connection(s)
I20260620 18:34:18.263927 12004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:18.266223 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12004
I20260620 18:34:18.266333 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:18.266561 10677 external_mini_cluster.cc:1428] Setting key f339628c9a6456be76004d8d43f5d273
I20260620 18:34:18.268211 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.256983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:52785 (local address 127.10.109.126:36109)
0620 18:34:18.257031 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.257034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.257214 (+ 180us) server_negotiation.cc:407] Connection header received
0620 18:34:18.257857 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.257859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.257874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.257943 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.258828 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.259689 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.260766 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.261127 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.261209 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.264385 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.264389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.264389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.264402 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.266494 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.267091 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.267092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.267093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.267157 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.267506 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.267506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.267507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.267641 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.267707 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.268038 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.268117 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:18.268419 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.256875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.66:52785)
0620 18:34:18.257079 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.257126 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.257722 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.258040 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.258044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.258407 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.258691 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.258698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.259787 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.259788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.260655 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.260657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.261246 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.261247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.261377 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.261942 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.261949 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.264188 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.266597 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.266605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.266608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.267008 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.267329 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.267330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.267330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.267429 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.267736 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.267742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.267893 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.268005 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.268179 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260620 18:34:18.268898 12135 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36109
I20260620 18:34:18.268980 12135 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.269169 12135 heartbeater.cc:511] Master 127.10.109.126:36109 requested a full tablet report, sending...
I20260620 18:34:18.269546 11791 ts_manager.cc:194] Registered new tserver with Master: 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66:36453)
I20260620 18:34:18.270045 11791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:52785
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:34:18.276679 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.269960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.66:47105)
0620 18:34:18.270839 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.270854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.270911 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.271039 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.271040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.271192 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.271246 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.271248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.272215 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.272216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.272711 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.272713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.273186 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.273187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.273241 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.273682 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.273687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.273973 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.275725 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.275726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.275727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.275910 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.276097 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.276098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.276098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.276155 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.276426 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.276427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.276465 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.276522 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.276595 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":82,"threads_started":1}
I20260620 18:34:18.276681 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.270023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:47105 (local address 127.10.109.126:36109)
0620 18:34:18.270179 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.270181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.270923 (+ 742us) server_negotiation.cc:407] Connection header received
0620 18:34:18.270951 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.270952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.270965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.271038 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.271316 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.272115 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.272792 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.273107 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.273180 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.274066 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.274067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.274068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.274075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.275635 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.275987 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.275988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.275988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.276036 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.276218 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.276218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.276219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.276361 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.276414 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.276566 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.276625 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:18.294593 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36109
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:18.376652 12144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.376859 12144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.376883 12144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.378055 12144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.378100 12144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.378123 12144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:18.378140 12144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:18.379673 12144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:36109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:18.379904 12144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.380144 12144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.385500 12150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.385752 12144 server_base.cc:1034] running on GCE node
W20260620 18:34:18.385500 12152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.385628 12149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.386189 12144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.386498 12144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.387669 12144 hybrid_clock.cc:648] HybridClock initialized: now 1781980458387663 us; error 41 us; skew 500 ppm
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:18.390676 12144 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:18.391170 12144 webserver.cc:460] Webserver started at http://127.10.109.67:36613/ using document root <none> and password file <none>
I20260620 18:34:18.391371 12144 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.391417 12144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.391505 12144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:18.392486 12144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "bd6737e1fb0048bcb3c73c388d68094e"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d911843e33a63725c13318e32a8f7d42"
server_key_iv: "3ba5572d32d57a328a68a6ee0ad018be"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.392789 12144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "bd6737e1fb0048bcb3c73c388d68094e"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d911843e33a63725c13318e32a8f7d42"
server_key_iv: "3ba5572d32d57a328a68a6ee0ad018be"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.394155 12144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:18.394922 12159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.395100 12144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:18.395195 12144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "bd6737e1fb0048bcb3c73c388d68094e"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-3jql"
server_key: "d911843e33a63725c13318e32a8f7d42"
server_key_iv: "3ba5572d32d57a328a68a6ee0ad018be"
server_key_version: "encryptionkey@0"
I20260620 18:34:18.395264 12144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.412767 12144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.415060 12144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.415221 12144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.415787 12144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.415838 12144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.415870 12144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.415899 12144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.422231 12144 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:46185
I20260620 18:34:18.422433 12274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:46185 every 8 connection(s)
I20260620 18:34:18.422618 12144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:18.426098 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.416268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:32891 (local address 127.10.109.126:36109)
0620 18:34:18.416343 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.416346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.416522 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:34:18.416976 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.416977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.416991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.417084 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.417954 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.418819 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.420336 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.420654 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.420740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.423321 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.423322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.423322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.423328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.424737 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.425190 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.425191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.425192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.425263 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.425527 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.425528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.425528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.425632 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.425687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.425965 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.426037 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:34:18.426308 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.416239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.67:32891)
0620 18:34:18.416379 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.416417 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.416884 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.417195 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.417198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.417541 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.417843 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.417850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.418917 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.418918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.420245 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.420248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.420741 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.420741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.420858 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.421360 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.421371 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.423242 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.424825 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.424831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.424833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.425062 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.425345 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.425345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.425346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.425443 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.425732 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.425737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.425847 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.425942 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.426062 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260620 18:34:18.426945 12275 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36109
I20260620 18:34:18.427021 12275 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.427225 12275 heartbeater.cc:511] Master 127.10.109.126:36109 requested a full tablet report, sending...
I20260620 18:34:18.427666 11791 ts_manager.cc:194] Registered new tserver with Master: bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185)
I20260620 18:34:18.428191 11791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:32891
I20260620 18:34:18.429638 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12144
I20260620 18:34:18.429742 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:18.429914 10677 external_mini_cluster.cc:1428] Setting key f33bae14198c1d0feb1932c900a55768
I20260620 18:34:18.430696 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:18.434504 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.428085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:34889 (local address 127.10.109.126:36109)
0620 18:34:18.428130 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.428131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.428210 (+ 79us) server_negotiation.cc:407] Connection header received
0620 18:34:18.428282 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.428283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.428293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.428349 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.428739 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.429324 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.430218 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.430649 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.430712 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.431634 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.431635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.431636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.431644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.433264 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.433700 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.433701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.433701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.433767 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.434014 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.434015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.434015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.434142 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.434201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.434372 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.434432 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:34:18.434612 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.428016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.10.109.67:34889)
0620 18:34:18.428149 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.428160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.428223 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.428352 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.428353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.428586 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.428663 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.428665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.429430 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.429431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.430135 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.430138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.430721 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.430722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.430819 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.431255 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.431259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.431513 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.433360 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.433362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.433362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.433598 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.433850 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.433851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.433852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.433929 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.434217 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.434218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.434270 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.434351 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.434463 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:18.454495 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.447936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.10.109.126:36109)
0620 18:34:18.447999 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.448002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.448011 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:18.448039 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.448040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.448053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.448134 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.448658 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.449148 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.449858 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.450044 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.450178 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.452413 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.452414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.452415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.452420 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.453575 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.453880 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.453880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.453880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.453926 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.454087 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.454087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.454087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.454206 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.454278 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.454381 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.454434 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:34:18.456207 11791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:18.457726 11791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:18.464403 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:46185 (local address 127.0.0.1:46460)
0620 18:34:18.460856 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.460864 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.460911 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.461111 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.461112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.461114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.461157 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.461158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.462345 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.462346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.463287 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.463289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.464057 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.464058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.464146 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.464165 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.464268 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.464322 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":58,"threads_started":1}
I20260620 18:34:18.464414 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.10.109.67:46185)
0620 18:34:18.460567 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.460572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.460965 (+ 393us) server_negotiation.cc:407] Connection header received
0620 18:34:18.461003 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.461005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.461026 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.461080 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.461257 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.462265 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.463377 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.463961 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.464057 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.464114 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.464285 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.464322 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260620 18:34:18.464572 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:36453 (local address 127.0.0.1:36668)
0620 18:34:18.460236 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.460273 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.460337 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.460794 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.460797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.460799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.460865 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.460867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.462143 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.462143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.463418 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.463421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.464338 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.464339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.464412 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.464416 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.464445 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.464493 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":60,"thread_start_us":138,"threads_started":1}
I20260620 18:34:18.464780 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.10.109.65:32811)
0620 18:34:18.460607 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.460611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.460620 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:18.460656 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.460658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.460672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.460771 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.461240 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.461923 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.463518 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.464238 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.464308 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.464362 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.464620 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.464664 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":81,"threads_started":1}
I20260620 18:34:18.464583 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:32811 (local address 127.0.0.1:55834)
0620 18:34:18.460422 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.460430 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.460501 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.461129 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.461130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.461131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.461166 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.461167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.462021 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.462022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.463433 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.463435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.464388 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.464389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.464443 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.464446 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.464490 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.464518 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":31,"thread_start_us":51,"threads_started":1}
I20260620 18:34:18.465224 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.460428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.10.109.66:36453)
0620 18:34:18.460609 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.460612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.460621 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:18.460680 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.460682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.460698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.460772 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.460953 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.462049 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.463518 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.464238 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.465001 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.465066 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.465090 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.465131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260620 18:34:18.465147 12209 tablet_service.cc:1467] Processing CreateTablet for tablet 200ff218c441483689481931f133d037 (DEFAULT_TABLE table=test-table [id=fe18990b085d4ab7af2672888a6e8a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.465512 12209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200ff218c441483689481931f133d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.465497 11929 tablet_service.cc:1467] Processing CreateTablet for tablet 200ff218c441483689481931f133d037 (DEFAULT_TABLE table=test-table [id=fe18990b085d4ab7af2672888a6e8a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.465811 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200ff218c441483689481931f133d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.465932 12069 tablet_service.cc:1467] Processing CreateTablet for tablet 200ff218c441483689481931f133d037 (DEFAULT_TABLE table=test-table [id=fe18990b085d4ab7af2672888a6e8a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.466223 12069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200ff218c441483689481931f133d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.469442 12299 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Bootstrap starting.
I20260620 18:34:18.469666 12298 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Bootstrap starting.
I20260620 18:34:18.469703 12300 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Bootstrap starting.
I20260620 18:34:18.470136 12299 tablet_bootstrap.cc:654] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.470460 12300 tablet_bootstrap.cc:654] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.470470 12299 log.cc:824] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.470564 12298 tablet_bootstrap.cc:654] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.470816 12300 log.cc:824] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.470902 12298 log.cc:824] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.471316 12299 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: No bootstrap required, opened a new log
I20260620 18:34:18.471386 12299 ts_tablet_manager.cc:1397] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:18.471540 12300 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: No bootstrap required, opened a new log
I20260620 18:34:18.471592 12298 tablet_bootstrap.cc:492] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: No bootstrap required, opened a new log
I20260620 18:34:18.471616 12300 ts_tablet_manager.cc:1397] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:18.471649 12298 ts_tablet_manager.cc:1397] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:34:18.472862 12299 raft_consensus.cc:348] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.473016 12299 raft_consensus.cc:374] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.473058 12299 raft_consensus.cc:729] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e78b8003128437fa699e8632280f37a, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.473186 12299 consensus_queue.cc:260] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.473462 11995 heartbeater.cc:503] Master 127.10.109.126:36109 was elected leader, sending a full tablet report...
I20260620 18:34:18.473538 12300 raft_consensus.cc:348] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.473537 12298 raft_consensus.cc:348] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.473690 12300 raft_consensus.cc:374] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.473690 12298 raft_consensus.cc:374] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.473454 12299 ts_tablet_manager.cc:1428] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:34:18.473724 12300 raft_consensus.cc:729] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1fc80acd1d4420ac10eb5697874af3, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.473724 12298 raft_consensus.cc:729] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6737e1fb0048bcb3c73c388d68094e, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.473826 12298 consensus_queue.cc:260] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.473826 12300 consensus_queue.cc:260] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.474080 12298 ts_tablet_manager.cc:1428] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:34:18.474080 12300 ts_tablet_manager.cc:1428] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:34:18.474118 12135 heartbeater.cc:503] Master 127.10.109.126:36109 was elected leader, sending a full tablet report...
I20260620 18:34:18.474198 12275 heartbeater.cc:503] Master 127.10.109.126:36109 was elected leader, sending a full tablet report...
W20260620 18:34:18.514608 12136 tablet.cc:2369] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:18.606092 11996 tablet.cc:2369] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:18.673208 12276 tablet.cc:2369] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:18.721832 12306 raft_consensus.cc:482] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:18.721982 12306 raft_consensus.cc:504] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.722467 12306 leader_election.cc:288] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185), 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811)
I20260620 18:34:18.725440 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.722787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:39397 (local address 127.10.109.67:46185)
0620 18:34:18.722834 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.722837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.722878 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:34:18.722908 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.722910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.722919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.723039 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.723206 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.723695 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.724715 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.725069 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.725125 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.725131 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.725325 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.725369 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:18.725464 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.722673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:46185 (local address 127.10.109.66:39397)
0620 18:34:18.722763 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.722778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.722828 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.723047 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.723048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.723050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.723129 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.723132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.723785 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.723786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.724616 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.724618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.725149 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.725151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.725248 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.725252 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.725314 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.725353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":20}
I20260620 18:34:18.725539 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.722802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:48831 (local address 127.10.109.65:32811)
0620 18:34:18.722860 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.722862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.722997 (+ 135us) server_negotiation.cc:407] Connection header received
0620 18:34:18.723029 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.723030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.723041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.723097 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.723288 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.723786 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.724755 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.725152 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.725233 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.725241 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.725420 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.725461 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:18.725697 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.722689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:32811 (local address 127.10.109.66:48831)
0620 18:34:18.722872 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.722882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.722942 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.723118 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.723119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.723120 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.723209 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.723211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.723872 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.723873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.724625 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.724628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.725238 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.725239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.725326 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.725331 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.725387 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.725424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":15,"thread_start_us":115,"threads_started":1}
I20260620 18:34:18.726068 12229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200ff218c441483689481931f133d037" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6737e1fb0048bcb3c73c388d68094e" is_pre_election: true
I20260620 18:34:18.726063 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200ff218c441483689481931f133d037" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a" is_pre_election: true
I20260620 18:34:18.726228 11934 raft_consensus.cc:2454] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 0.
I20260620 18:34:18.726213 12229 raft_consensus.cc:2454] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 0.
I20260620 18:34:18.726444 12021 leader_election.cc:302] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, bd6737e1fb0048bcb3c73c388d68094e; no voters:
I20260620 18:34:18.726598 12306 raft_consensus.cc:2781] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:18.726671 12306 raft_consensus.cc:482] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:18.726694 12306 raft_consensus.cc:3037] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.727778 12306 raft_consensus.cc:504] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.727986 12306 leader_election.cc:288] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 election: Requested vote from peers bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185), 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811)
I20260620 18:34:18.728060 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200ff218c441483689481931f133d037" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a"
I20260620 18:34:18.728122 11934 raft_consensus.cc:3037] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.728760 11934 raft_consensus.cc:2454] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 1.
I20260620 18:34:18.728842 12229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200ff218c441483689481931f133d037" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6737e1fb0048bcb3c73c388d68094e"
I20260620 18:34:18.728933 12020 leader_election.cc:302] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, 8e78b8003128437fa699e8632280f37a; no voters:
I20260620 18:34:18.728925 12229 raft_consensus.cc:3037] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.729152 12306 raft_consensus.cc:2781] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:18.729278 12306 raft_consensus.cc:686] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1fc80acd1d4420ac10eb5697874af3, State: Running, Role: LEADER
I20260620 18:34:18.729341 12306 consensus_queue.cc:237] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } }
I20260620 18:34:18.729720 12229 raft_consensus.cc:2454] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 1.
I20260620 18:34:18.730213 11792 catalog_manager.cc:5495] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:18.746767 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.744471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.10.109.66:36453)
0620 18:34:18.744526 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.744528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.744541 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:34:18.744625 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.744626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.744640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.744697 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:18.744805 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.745262 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.745971 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.746173 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.746222 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.746375 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.746454 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.746558 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.746580 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.746597 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:18.748577 12229 raft_consensus.cc:1261] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Refusing update from remote peer 5f1fc80acd1d4420ac10eb5697874af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.748589 11934 raft_consensus.cc:1261] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Refusing update from remote peer 5f1fc80acd1d4420ac10eb5697874af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.748912 12306 consensus_queue.cc:1035] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.749083 12308 consensus_queue.cc:1035] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.751214 12311 mvcc.cc:205] Tried to move back new op lower bound from 7298991959031201792 to 7298991958955864064. Current Snapshot: MvccSnapshot[applied={T|T < 7298991959031201792}]
I20260620 18:34:18.751972 12313 mvcc.cc:205] Tried to move back new op lower bound from 7298991959031201792 to 7298991958955864064. Current Snapshot: MvccSnapshot[applied={T|T < 7298991959031201792}]
I20260620 18:34:18.755015 11792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:18.755120 11792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:18.755908 11792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Sending DeleteTablet for 3 replicas of tablet 200ff218c441483689481931f133d037
I20260620 18:34:18.756268 11929 tablet_service.cc:1514] Processing DeleteTablet for tablet 200ff218c441483689481931f133d037 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55834
I20260620 18:34:18.756278 12209 tablet_service.cc:1514] Processing DeleteTablet for tablet 200ff218c441483689481931f133d037 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:46460
I20260620 18:34:18.756274 12069 tablet_service.cc:1514] Processing DeleteTablet for tablet 200ff218c441483689481931f133d037 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36668
I20260620 18:34:18.756605 12321 tablet_replica.cc:330] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: stopping tablet replica
I20260620 18:34:18.756649 12323 tablet_replica.cc:330] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: stopping tablet replica
I20260620 18:34:18.756742 12321 raft_consensus.cc:2229] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:18.756744 12323 raft_consensus.cc:2229] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:18.756767 11792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:34:18.756850 12323 raft_consensus.cc:2258] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:34:18.756904 11792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:18.757040 12321 raft_consensus.cc:2258] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:18.757289 12323 ts_tablet_manager.cc:1905] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:18.757548 12321 ts_tablet_manager.cc:1905] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:18.758488 11929 tablet_service.cc:1467] Processing CreateTablet for tablet c825ba9f276c449d933d1fee82ab7b09 (DEFAULT_TABLE table=test-table [id=5290b849087c47b8acd4728b15f76392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.758620 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c825ba9f276c449d933d1fee82ab7b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.758884 12323 ts_tablet_manager.cc:1918] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:18.758953 12323 log.cc:1196] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/200ff218c441483689481931f133d037
I20260620 18:34:18.759233 12323 ts_tablet_manager.cc:1939] T 200ff218c441483689481931f133d037 P bd6737e1fb0048bcb3c73c388d68094e: Deleting consensus metadata
I20260620 18:34:18.759371 12321 ts_tablet_manager.cc:1918] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:18.759377 12322 tablet_replica.cc:330] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: stopping tablet replica
I20260620 18:34:18.759472 12321 log.cc:1196] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/200ff218c441483689481931f133d037
I20260620 18:34:18.759485 12322 raft_consensus.cc:2229] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:18.759595 12322 raft_consensus.cc:2258] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:18.759783 12321 ts_tablet_manager.cc:1939] T 200ff218c441483689481931f133d037 P 8e78b8003128437fa699e8632280f37a: Deleting consensus metadata
I20260620 18:34:18.759912 12209 tablet_service.cc:1467] Processing CreateTablet for tablet c825ba9f276c449d933d1fee82ab7b09 (DEFAULT_TABLE table=test-table [id=5290b849087c47b8acd4728b15f76392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.759970 11777 catalog_manager.cc:4841] TS bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185): tablet 200ff218c441483689481931f133d037 (table test-table [id=fe18990b085d4ab7af2672888a6e8a7a]) successfully deleted
I20260620 18:34:18.760036 12209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c825ba9f276c449d933d1fee82ab7b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.760324 12322 ts_tablet_manager.cc:1905] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:18.760974 12298 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Bootstrap starting.
I20260620 18:34:18.761189 11776 catalog_manager.cc:4841] TS 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811): tablet 200ff218c441483689481931f133d037 (table test-table [id=fe18990b085d4ab7af2672888a6e8a7a]) successfully deleted
I20260620 18:34:18.761320 12069 tablet_service.cc:1467] Processing CreateTablet for tablet c825ba9f276c449d933d1fee82ab7b09 (DEFAULT_TABLE table=test-table [id=5290b849087c47b8acd4728b15f76392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.761377 12298 tablet_bootstrap.cc:654] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.761410 12069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c825ba9f276c449d933d1fee82ab7b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.761969 12298 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: No bootstrap required, opened a new log
I20260620 18:34:18.762022 12298 ts_tablet_manager.cc:1397] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:18.762161 12298 raft_consensus.cc:348] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.762226 12298 raft_consensus.cc:374] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.762261 12298 raft_consensus.cc:729] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6737e1fb0048bcb3c73c388d68094e, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.762307 12298 consensus_queue.cc:260] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.762401 12298 ts_tablet_manager.cc:1428] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:18.763069 12299 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Bootstrap starting.
I20260620 18:34:18.763221 12300 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Bootstrap starting.
I20260620 18:34:18.763408 12299 tablet_bootstrap.cc:654] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.763589 12300 tablet_bootstrap.cc:654] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.763975 12322 ts_tablet_manager.cc:1918] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:18.763988 12299 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: No bootstrap required, opened a new log
I20260620 18:34:18.764042 12322 log.cc:1196] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/200ff218c441483689481931f133d037
I20260620 18:34:18.764050 12299 ts_tablet_manager.cc:1397] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:18.764176 12299 raft_consensus.cc:348] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.764253 12299 raft_consensus.cc:374] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.764281 12299 raft_consensus.cc:729] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e78b8003128437fa699e8632280f37a, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.764297 12322 ts_tablet_manager.cc:1939] T 200ff218c441483689481931f133d037 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting consensus metadata
I20260620 18:34:18.764320 12299 consensus_queue.cc:260] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.764433 12299 ts_tablet_manager.cc:1428] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:18.764642 11777 catalog_manager.cc:4841] TS 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66:36453): tablet 200ff218c441483689481931f133d037 (table test-table [id=fe18990b085d4ab7af2672888a6e8a7a]) successfully deleted
I20260620 18:34:18.764849 12300 tablet_bootstrap.cc:492] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: No bootstrap required, opened a new log
I20260620 18:34:18.764896 12300 ts_tablet_manager.cc:1397] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:18.765023 12300 raft_consensus.cc:348] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.765102 12300 raft_consensus.cc:374] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.765123 12300 raft_consensus.cc:729] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1fc80acd1d4420ac10eb5697874af3, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.765166 12300 consensus_queue.cc:260] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.765247 12300 ts_tablet_manager.cc:1428] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.821102 12305 raft_consensus.cc:482] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:18.821231 12305 raft_consensus.cc:504] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.821580 12305 leader_election.cc:288] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811), 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66:36453)
I20260620 18:34:18.824813 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.821845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:60983 (local address 127.10.109.66:36453)
0620 18:34:18.821890 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.821892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.822090 (+ 198us) server_negotiation.cc:407] Connection header received
0620 18:34:18.822162 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.822163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.822172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.822216 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.822373 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.823063 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824007 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.824436 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824507 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.824515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.824713 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.824747 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:18.824836 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.821757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:36453 (local address 127.10.109.67:60983)
0620 18:34:18.822033 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.822043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.822098 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.822230 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.822230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.822232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.822295 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.822296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.823159 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.823160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.823913 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.823915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.824538 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.824626 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.824630 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.824694 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.824733 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":159,"thread_start_us":63,"threads_started":1}
I20260620 18:34:18.825122 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c825ba9f276c449d933d1fee82ab7b09" candidate_uuid: "bd6737e1fb0048bcb3c73c388d68094e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1fc80acd1d4420ac10eb5697874af3" is_pre_election: true
I20260620 18:34:18.825181 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.821754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:32811 (local address 127.10.109.67:56487)
0620 18:34:18.821817 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.821829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.821869 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.822046 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.822047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.822049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.822126 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.822127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.823117 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.823118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.824251 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.824253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.824943 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.825022 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.825025 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.825091 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.825124 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:34:18.825227 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.821859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:56487 (local address 127.10.109.65:32811)
0620 18:34:18.821909 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.821911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.821919 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:18.821947 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.821949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.821958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.822021 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.822194 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.823021 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824346 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.824851 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.824920 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.824928 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.825118 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.825166 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 18:34:18.825229 12089 raft_consensus.cc:2454] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6737e1fb0048bcb3c73c388d68094e in term 0.
I20260620 18:34:18.825392 12161 leader_election.cc:302] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, bd6737e1fb0048bcb3c73c388d68094e; no voters:
I20260620 18:34:18.825507 12305 raft_consensus.cc:2781] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:18.825547 12305 raft_consensus.cc:482] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:18.825567 12305 raft_consensus.cc:3037] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.825611 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c825ba9f276c449d933d1fee82ab7b09" candidate_uuid: "bd6737e1fb0048bcb3c73c388d68094e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a" is_pre_election: true
I20260620 18:34:18.825688 11934 raft_consensus.cc:2454] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6737e1fb0048bcb3c73c388d68094e in term 0.
I20260620 18:34:18.826385 12305 raft_consensus.cc:504] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.826519 12305 leader_election.cc:288] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [CANDIDATE]: Term 1 election: Requested vote from peers 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811), 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66:36453)
I20260620 18:34:18.826704 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c825ba9f276c449d933d1fee82ab7b09" candidate_uuid: "bd6737e1fb0048bcb3c73c388d68094e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a"
I20260620 18:34:18.826726 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c825ba9f276c449d933d1fee82ab7b09" candidate_uuid: "bd6737e1fb0048bcb3c73c388d68094e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1fc80acd1d4420ac10eb5697874af3"
I20260620 18:34:18.826804 11934 raft_consensus.cc:3037] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.826812 12089 raft_consensus.cc:3037] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.827544 12089 raft_consensus.cc:2454] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6737e1fb0048bcb3c73c388d68094e in term 1.
I20260620 18:34:18.827684 12161 leader_election.cc:302] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, bd6737e1fb0048bcb3c73c388d68094e; no voters:
I20260620 18:34:18.827759 12305 raft_consensus.cc:2781] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:18.827879 12305 raft_consensus.cc:686] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 1 LEADER]: Becoming Leader. State: Replica: bd6737e1fb0048bcb3c73c388d68094e, State: Running, Role: LEADER
I20260620 18:34:18.827951 12305 consensus_queue.cc:237] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.828016 11934 raft_consensus.cc:2454] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6737e1fb0048bcb3c73c388d68094e in term 1.
I20260620 18:34:18.828694 11792 catalog_manager.cc:5495] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67). New cstate: current_term: 1 leader_uuid: "bd6737e1fb0048bcb3c73c388d68094e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } health_report { overall_health: HEALTHY } } }
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:18.868256 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.860026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.10.109.126:36109)
0620 18:34:18.860185 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.860187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.860197 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:18.860253 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.860255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.860268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.860373 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.860907 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.861673 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.862682 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.862986 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.863024 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.865451 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.865452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.865453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.865459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.866979 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.867426 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.867427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.867428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.867481 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.867666 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.867666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.867667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.867775 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.867824 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.868058 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.868141 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 18:34:18.868569 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.859893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36109 (local address 127.0.0.1:57826)
0620 18:34:18.860078 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.860089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.860145 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.860386 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.860387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.860755 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.860819 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.860821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.861760 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.861760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.862595 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.862597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.863127 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.863128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.863176 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.863718 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.863724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.865358 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.867066 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.867073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.867075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.867333 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.867549 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.867550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.867551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.867606 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.867872 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.867879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.867928 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.868364 (+ 436us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.868468 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260620 18:34:18.872808 11792 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:57826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:34:18.873019 11792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:18.874852 12069 tablet_service.cc:1467] Processing CreateTablet for tablet ea8c8f1774dd46039539bf996afee11b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f25b99940e944f985e80a673b6635c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:18.874869 12209 tablet_service.cc:1467] Processing CreateTablet for tablet ea8c8f1774dd46039539bf996afee11b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f25b99940e944f985e80a673b6635c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:18.874895 11929 tablet_service.cc:1467] Processing CreateTablet for tablet ea8c8f1774dd46039539bf996afee11b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f25b99940e944f985e80a673b6635c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:18.875008 12209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8c8f1774dd46039539bf996afee11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.875005 12069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8c8f1774dd46039539bf996afee11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.875072 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8c8f1774dd46039539bf996afee11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.876418 12298 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: Bootstrap starting.
I20260620 18:34:18.876418 12300 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: Bootstrap starting.
I20260620 18:34:18.876765 12300 tablet_bootstrap.cc:654] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.876780 12298 tablet_bootstrap.cc:654] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.877034 12299 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: Bootstrap starting.
I20260620 18:34:18.877457 12299 tablet_bootstrap.cc:654] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.877506 12298 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: No bootstrap required, opened a new log
I20260620 18:34:18.877545 12298 ts_tablet_manager.cc:1397] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:18.877815 12298 raft_consensus.cc:348] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.877894 12298 raft_consensus.cc:374] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.877920 12298 raft_consensus.cc:729] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6737e1fb0048bcb3c73c388d68094e, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.877969 12298 consensus_queue.cc:260] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.878055 12298 ts_tablet_manager.cc:1428] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:18.878126 12299 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: No bootstrap required, opened a new log
I20260620 18:34:18.878175 12299 ts_tablet_manager.cc:1397] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:18.878160 12305 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.878217 12305 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.878315 12299 raft_consensus.cc:348] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.878387 12299 raft_consensus.cc:374] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.878407 12299 raft_consensus.cc:729] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e78b8003128437fa699e8632280f37a, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.878443 12299 consensus_queue.cc:260] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.878520 12299 ts_tablet_manager.cc:1428] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:18.878675 12304 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.878774 12304 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.878854 12300 tablet_bootstrap.cc:492] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: No bootstrap required, opened a new log
I20260620 18:34:18.878902 12300 ts_tablet_manager.cc:1397] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:18.879002 12300 raft_consensus.cc:348] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.879060 12300 raft_consensus.cc:374] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.879078 12300 raft_consensus.cc:729] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1fc80acd1d4420ac10eb5697874af3, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.879148 12300 consensus_queue.cc:260] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.879240 12300 ts_tablet_manager.cc:1428] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.879256 12316 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.879343 12316 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.942533 12316 raft_consensus.cc:482] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:18.942627 12316 raft_consensus.cc:504] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.942827 12316 leader_election.cc:288] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811), bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185)
I20260620 18:34:18.943013 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8c8f1774dd46039539bf996afee11b" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a" is_pre_election: true
I20260620 18:34:18.943040 12229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8c8f1774dd46039539bf996afee11b" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6737e1fb0048bcb3c73c388d68094e" is_pre_election: true
I20260620 18:34:18.943125 12229 raft_consensus.cc:2454] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 0.
I20260620 18:34:18.943125 11934 raft_consensus.cc:2454] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 0.
I20260620 18:34:18.943271 12020 leader_election.cc:302] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, 8e78b8003128437fa699e8632280f37a; no voters:
I20260620 18:34:18.943390 12316 raft_consensus.cc:2781] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:18.943442 12316 raft_consensus.cc:482] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:18.943480 12316 raft_consensus.cc:3037] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.944173 12316 raft_consensus.cc:504] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.944319 12316 leader_election.cc:288] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 election: Requested vote from peers 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811), bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185)
I20260620 18:34:18.944458 12229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8c8f1774dd46039539bf996afee11b" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6737e1fb0048bcb3c73c388d68094e"
I20260620 18:34:18.944489 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8c8f1774dd46039539bf996afee11b" candidate_uuid: "5f1fc80acd1d4420ac10eb5697874af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78b8003128437fa699e8632280f37a"
I20260620 18:34:18.944530 12229 raft_consensus.cc:3037] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.944554 11934 raft_consensus.cc:3037] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.945257 12229 raft_consensus.cc:2454] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 1.
I20260620 18:34:18.945257 11934 raft_consensus.cc:2454] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1fc80acd1d4420ac10eb5697874af3 in term 1.
I20260620 18:34:18.945452 12021 leader_election.cc:302] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1fc80acd1d4420ac10eb5697874af3, bd6737e1fb0048bcb3c73c388d68094e; no voters:
I20260620 18:34:18.945559 12316 raft_consensus.cc:2781] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:18.945649 12316 raft_consensus.cc:686] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1fc80acd1d4420ac10eb5697874af3, State: Running, Role: LEADER
I20260620 18:34:18.945710 12316 consensus_queue.cc:237] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } }
I20260620 18:34:18.945966 12306 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: TxnStatusTablet state changed. Reason: New leader 5f1fc80acd1d4420ac10eb5697874af3. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.946062 12306 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:18.946193 12340 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:34:18.946331 12340 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:34:18.946447 11790 catalog_manager.cc:5495] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:18.955535 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.953084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:36453 (local address 127.0.0.1:36698)
0620 18:34:18.953165 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.953178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.953218 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.953361 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.953362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.953363 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:18.953409 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.953411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.954028 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.954029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.954665 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.954667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.955077 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.955078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.955146 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.955160 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.955365 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.955374 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.955427 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.955474 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:34:18.955550 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.953135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.10.109.66:36453)
0620 18:34:18.953189 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.953191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.953215 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:34:18.953290 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.953291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.953302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.953354 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:18.953473 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.953939 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.954780 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.954986 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.955074 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.955256 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.955291 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.955363 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.955460 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.955486 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:34:18.956384 11934 raft_consensus.cc:1261] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Refusing update from remote peer 5f1fc80acd1d4420ac10eb5697874af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.956382 12229 raft_consensus.cc:1261] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Refusing update from remote peer 5f1fc80acd1d4420ac10eb5697874af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.956596 12306 consensus_queue.cc:1035] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.956734 12316 consensus_queue.cc:1035] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.957070 12305 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: TxnStatusTablet state changed. Reason: New leader 5f1fc80acd1d4420ac10eb5697874af3. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.957162 12305 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.957237 12304 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: TxnStatusTablet state changed. Reason: New leader 5f1fc80acd1d4420ac10eb5697874af3. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.957325 12304 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.957908 12306 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.957989 12306 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:18.958297 12305 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.958380 12305 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P bd6737e1fb0048bcb3c73c388d68094e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.959259 12306 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.959342 12306 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 5f1fc80acd1d4420ac10eb5697874af3: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:18.959407 12304 tablet_replica.cc:439] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f1fc80acd1d4420ac10eb5697874af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 } } peers { permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 } } peers { permanent_uuid: "bd6737e1fb0048bcb3c73c388d68094e" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 46185 } } }
I20260620 18:34:18.959434 12312 mvcc.cc:205] Tried to move back new op lower bound from 7298991959883268096 to 7298991959841296384. Current Snapshot: MvccSnapshot[applied={T|T < 7298991959883268096}]
I20260620 18:34:18.959478 12304 tablet_replica.cc:442] T ea8c8f1774dd46039539bf996afee11b P 8e78b8003128437fa699e8632280f37a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:18.962424 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.959787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.10.109.67:46185)
0620 18:34:18.959823 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.959825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.959833 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:18.959909 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.959910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.959928 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.960005 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:18.960160 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.960851 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.961548 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.961843 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.961911 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.962064 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.962180 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.962253 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.962342 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.962359 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:34:18.967309 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.964643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:34077 (local address 127.10.109.66:36453)
0620 18:34:18.964706 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.964708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.964859 (+ 151us) server_negotiation.cc:407] Connection header received
0620 18:34:18.964903 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.964904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.964914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.964974 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:18.965053 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.965713 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.966484 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.966816 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.966883 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.967097 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.967127 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.967180 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.967229 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.967246 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:34:18.967324 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.964580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:36453 (local address 127.10.109.67:34077)
0620 18:34:18.964777 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.964788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.964848 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.964975 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.964975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.964976 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:18.965013 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.965014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.965805 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.965805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.966393 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.966395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.966920 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.966922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.966983 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.967004 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.967189 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.967191 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.967226 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.967259 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260620 18:34:18.969681 12089 raft_consensus.cc:1261] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Refusing update from remote peer bd6737e1fb0048bcb3c73c388d68094e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.969689 11934 raft_consensus.cc:1261] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Refusing update from remote peer bd6737e1fb0048bcb3c73c388d68094e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:18.969921 12328 consensus_queue.cc:1035] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e78b8003128437fa699e8632280f37a" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.970126 12305 consensus_queue.cc:1035] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1fc80acd1d4420ac10eb5697874af3" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:18.977944 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.975052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:46185 (local address 127.10.109.66:50769)
0620 18:34:18.975223 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.975232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.975281 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.975398 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.975398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.975399 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:18.975432 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.975433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.976249 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.976250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.977172 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.977175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.977601 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.977602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.977643 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.977655 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.977824 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.977826 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.977859 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.977891 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260620 18:34:18.977944 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.975114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:50769 (local address 127.10.109.67:46185)
0620 18:34:18.975147 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.975149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.975299 (+ 150us) server_negotiation.cc:407] Connection header received
0620 18:34:18.975325 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.975325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.975335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.975398 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:18.975474 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.976170 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.977246 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.977522 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.977589 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.977735 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:18.977765 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:18.977817 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.977862 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.977880 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:34:18.987686 11790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:18.987783 11790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:18.988379 11790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f5ecb8b20584d59a593d9d250e3dfee: Sending DeleteTablet for 3 replicas of tablet c825ba9f276c449d933d1fee82ab7b09
I20260620 18:34:18.988660 12209 tablet_service.cc:1514] Processing DeleteTablet for tablet c825ba9f276c449d933d1fee82ab7b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:46460
I20260620 18:34:18.988668 11929 tablet_service.cc:1514] Processing DeleteTablet for tablet c825ba9f276c449d933d1fee82ab7b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55834
I20260620 18:34:18.988744 12323 tablet_replica.cc:330] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: stopping tablet replica
I20260620 18:34:18.988757 12321 tablet_replica.cc:330] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: stopping tablet replica
I20260620 18:34:18.988788 12323 raft_consensus.cc:2229] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:18.988818 12321 raft_consensus.cc:2229] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:18.988852 12321 raft_consensus.cc:2258] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:18.988862 12323 raft_consensus.cc:2258] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:18.989142 12069 tablet_service.cc:1514] Processing DeleteTablet for tablet c825ba9f276c449d933d1fee82ab7b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:18 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36668
I20260620 18:34:18.989229 12323 ts_tablet_manager.cc:1905] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:18.989259 12322 tablet_replica.cc:330] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: stopping tablet replica
I20260620 18:34:18.989307 12322 raft_consensus.cc:2229] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:18.989336 12322 raft_consensus.cc:2258] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:18.989609 12322 ts_tablet_manager.cc:1905] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:18.990871 12323 ts_tablet_manager.cc:1918] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:18.990929 12323 log.cc:1196] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/c825ba9f276c449d933d1fee82ab7b09
I20260620 18:34:18.991142 12323 ts_tablet_manager.cc:1939] T c825ba9f276c449d933d1fee82ab7b09 P bd6737e1fb0048bcb3c73c388d68094e: Deleting consensus metadata
I20260620 18:34:18.991237 12322 ts_tablet_manager.cc:1918] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:18.991288 12322 log.cc:1196] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/c825ba9f276c449d933d1fee82ab7b09
I20260620 18:34:18.991515 12322 ts_tablet_manager.cc:1939] T c825ba9f276c449d933d1fee82ab7b09 P 5f1fc80acd1d4420ac10eb5697874af3: Deleting consensus metadata
I20260620 18:34:18.991607 11777 catalog_manager.cc:4841] TS bd6737e1fb0048bcb3c73c388d68094e (127.10.109.67:46185): tablet c825ba9f276c449d933d1fee82ab7b09 (table test-table [id=5290b849087c47b8acd4728b15f76392]) successfully deleted
I20260620 18:34:18.992188 11777 catalog_manager.cc:4841] TS 5f1fc80acd1d4420ac10eb5697874af3 (127.10.109.66:36453): tablet c825ba9f276c449d933d1fee82ab7b09 (table test-table [id=5290b849087c47b8acd4728b15f76392]) successfully deleted
I20260620 18:34:18.998965 12321 ts_tablet_manager.cc:1905] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.000527 12321 ts_tablet_manager.cc:1918] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:19.000581 12321 log.cc:1196] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/c825ba9f276c449d933d1fee82ab7b09
I20260620 18:34:19.000782 12321 ts_tablet_manager.cc:1939] T c825ba9f276c449d933d1fee82ab7b09 P 8e78b8003128437fa699e8632280f37a: Deleting consensus metadata
Jun 20 18:34:18 dist-test-slave-3jql krb5kdc[11747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:19.001271 11776 catalog_manager.cc:4841] TS 8e78b8003128437fa699e8632280f37a (127.10.109.65:32811): tablet c825ba9f276c449d933d1fee82ab7b09 (table test-table [id=5290b849087c47b8acd4728b15f76392]) successfully deleted
I20260620 18:34:19.004837 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.989600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.10.109.65:32811)
0620 18:34:18.989639 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.989640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.994955 (+ 5315us) server_negotiation.cc:407] Connection header received
0620 18:34:18.995011 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.995012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.995025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.995095 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.995736 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.996482 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.997567 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.997909 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.997981 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.001638 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.001646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.001647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:19.001654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.003604 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.004000 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.004003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.004004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.004067 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.004281 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.004281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.004282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.004439 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.004583 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.004662 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.004741 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 18:34:19.005321 11988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838
I20260620 18:34:19.013700 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.006224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.10.109.126:36109)
0620 18:34:19.006260 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.006262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.006428 (+ 166us) server_negotiation.cc:407] Connection header received
0620 18:34:19.006497 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.006498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.006508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.006566 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.007202 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.007973 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.009017 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.009333 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.009403 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.010789 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.010790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.010790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:19.010797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.012478 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.012873 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.012874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.012874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.012932 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.013167 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.013168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.013168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.013294 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.013390 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.013533 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.013607 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 18:34:19.013983 11790 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834
I20260620 18:34:19.014636 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11864
I20260620 18:34:19.022341 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12004
I20260620 18:34:19.028606 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12144
I20260620 18:34:19.034597 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 11760
2026-06-20T18:34:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2402 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:19 dist-test-slave-3jql krb5kdc[12359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:19 dist-test-slave-3jql krb5kdc[12359](info): set up 2 sockets
Jun 20 18:34:19 dist-test-slave-3jql krb5kdc[12359](info): commencing operation
krb5kdc: starting...
W20260620 18:34:20.075269 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:20.193542 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40149
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:20.273757 12372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:20.273988 12372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:20.274022 12372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:20.275200 12372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:20.275260 12372 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:34:20.275283 12372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:20.275310 12372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:20.275342 12372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:20.275360 12372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:20.276780 12372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:40149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:20.277040 12372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:20.277271 12372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:20.282485 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.282498 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:20.282532 12372 server_base.cc:1034] running on GCE node
W20260620 18:34:20.282500 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:20.283030 12372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:20.283324 12372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:20.284511 12372 hybrid_clock.cc:648] HybridClock initialized: now 1781980460284482 us; error 61 us; skew 500 ppm
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:20.287233 12372 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:20.287717 12372 webserver.cc:460] Webserver started at http://127.10.109.126:44393/ using document root <none> and password file <none>
I20260620 18:34:20.287977 12372 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:20.288055 12372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:20.288199 12372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:20.289249 12372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "f67575e68329429d86e1a823c80636e8"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "54f64a6e539d3d0489f4e84d0d521905"
server_key_iv: "49d160bebf905ee71ea1880c3975111e"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.289594 12372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "f67575e68329429d86e1a823c80636e8"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "54f64a6e539d3d0489f4e84d0d521905"
server_key_iv: "49d160bebf905ee71ea1880c3975111e"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.291355 12372 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 18:34:20.292080 12387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.292304 12372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:20.292411 12372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "f67575e68329429d86e1a823c80636e8"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "54f64a6e539d3d0489f4e84d0d521905"
server_key_iv: "49d160bebf905ee71ea1880c3975111e"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.292493 12372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:20.302400 12372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:20.304277 12372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:20.304435 12372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:20.309183 12372 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:40149
I20260620 18:34:20.309247 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:40149 every 8 connection(s)
I20260620 18:34:20.309526 12372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:20.310118 12450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.312264 12450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Bootstrap starting.
I20260620 18:34:20.312770 12450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.312991 12450 log.cc:824] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:20.314033 12450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: No bootstrap required, opened a new log
I20260620 18:34:20.316181 12450 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } }
I20260620 18:34:20.316313 12450 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.316335 12450 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67575e68329429d86e1a823c80636e8, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.316469 12450 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } }
I20260620 18:34:20.316550 12450 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:20.316572 12450 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:20.316599 12450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.317090 12450 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } }
I20260620 18:34:20.317178 12450 leader_election.cc:302] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67575e68329429d86e1a823c80636e8; no voters:
I20260620 18:34:20.317322 12450 leader_election.cc:288] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:20.317346 12453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:20.317479 12453 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [term 1 LEADER]: Becoming Leader. State: Replica: f67575e68329429d86e1a823c80636e8, State: Running, Role: LEADER
I20260620 18:34:20.317567 12453 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } }
I20260620 18:34:20.317601 12450 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:20.317975 12454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67575e68329429d86e1a823c80636e8. Latest consensus state: current_term: 1 leader_uuid: "f67575e68329429d86e1a823c80636e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } } }
I20260620 18:34:20.318135 12454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:20.318154 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12372
I20260620 18:34:20.318102 12455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67575e68329429d86e1a823c80636e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67575e68329429d86e1a823c80636e8" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40149 } } }
I20260620 18:34:20.318239 12455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:20.318257 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:20.318491 10677 external_mini_cluster.cc:1428] Setting key 7edc604479b7172ea3dec2672778332f
I20260620 18:34:20.318493 12457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:20.319028 12457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:20.320632 12457 catalog_manager.cc:1347] Generated new cluster ID: 7f14c343fca344b8bcd4aa8a6786dc70
I20260620 18:34:20.320688 12457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.328843 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.319383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.10.109.126:40149)
0620 18:34:20.319541 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.319549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.319579 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:34:20.319688 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.319695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.319746 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.320612 (+ 866us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.320705 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.321620 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.322378 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.322845 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.322970 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.325169 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.325173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.325173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.325190 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.327344 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.327718 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.327720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.327721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.327815 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.328028 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.328029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.328030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.328261 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.328449 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.328512 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.328660 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260620 18:34:20.332561 12457 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:20.333223 12457 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:20.340960 12457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Generated new TSK 0
I20260620 18:34:20.341269 12457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:20.369594 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40149
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:20.454355 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:20.454546 12476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:20.454571 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:20.455761 12476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:20.455803 12476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:20.455823 12476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:34:20.455835 12476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:20.455895 12476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:20.457353 12476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:20.457603 12476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:20.457846 12476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:20.462718 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.462725 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.462833 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:20.462934 12476 server_base.cc:1034] running on GCE node
I20260620 18:34:20.463127 12476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:20.463382 12476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:20.464571 12476 hybrid_clock.cc:648] HybridClock initialized: now 1781980460464546 us; error 24 us; skew 500 ppm
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:20.467042 12476 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:20.467465 12476 webserver.cc:460] Webserver started at http://127.10.109.65:44547/ using document root <none> and password file <none>
I20260620 18:34:20.467661 12476 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:20.467701 12476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:20.467787 12476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:20.468727 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "4b49a46884144c94973980f1b8a39009"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "57c4abafc3c2d359811fba0b26c48925"
server_key_iv: "fbcff700f8816c87004b63ed795c0557"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.469004 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "4b49a46884144c94973980f1b8a39009"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "57c4abafc3c2d359811fba0b26c48925"
server_key_iv: "fbcff700f8816c87004b63ed795c0557"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.470232 12476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:20.471040 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.471201 12476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:20.471280 12476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "4b49a46884144c94973980f1b8a39009"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "57c4abafc3c2d359811fba0b26c48925"
server_key_iv: "fbcff700f8816c87004b63ed795c0557"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.471331 12476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:20.497874 12476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:20.500269 12476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:20.500380 12476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:20.500824 12476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:20.500857 12476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.500880 12476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:20.500892 12476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.507949 12476 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:37095
I20260620 18:34:20.508051 12606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:37095 every 8 connection(s)
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.508773 12476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:20.512012 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.501523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:54733 (local address 127.10.109.126:40149)
0620 18:34:20.501575 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.501577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.501805 (+ 228us) server_negotiation.cc:407] Connection header received
0620 18:34:20.502489 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.502491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.502506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.502581 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.503478 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.504300 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.506094 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.506438 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.506516 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.509044 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.509045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.509045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.509053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.510551 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.511027 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.511029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.511029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.511095 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.511353 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.511354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.511354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.511471 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.511565 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.511835 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.511925 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:20.512177 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.501515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.65:54733)
0620 18:34:20.501685 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.501718 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.502353 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.502695 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.502699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.503115 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.503388 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.503394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.504395 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.504396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.505971 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.505974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.506556 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.506557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.506764 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.507333 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.507343 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.508941 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.510659 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.510665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.510667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.510948 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.511165 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.511166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.511167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.511278 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.511557 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.511563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.511698 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.511799 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.511948 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260620 18:34:20.512693 12607 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40149
I20260620 18:34:20.512773 12607 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:20.512950 12607 heartbeater.cc:511] Master 127.10.109.126:40149 requested a full tablet report, sending...
I20260620 18:34:20.513415 12404 ts_manager.cc:194] Registered new tserver with Master: 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095)
I20260620 18:34:20.514195 12404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:54733
I20260620 18:34:20.514786 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12476
I20260620 18:34:20.514874 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:20.515066 10677 external_mini_cluster.cc:1428] Setting key 7dee8185e9e8f973ab3590210ceea30f
I20260620 18:34:20.520191 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.513800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:35365 (local address 127.10.109.126:40149)
0620 18:34:20.513840 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.513842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.513913 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:34:20.513978 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.513979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.513987 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.514033 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.514314 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.515043 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.515894 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.516213 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.516285 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.517298 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.517299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.517299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.517305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.518867 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.519273 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.519273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.519274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.519323 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.519586 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.519587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.519587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.519749 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.519813 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.519987 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.520134 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:20.520241 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.513742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.65:35365)
0620 18:34:20.513856 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.513865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.513924 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.514035 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.514036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.514173 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.514218 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.514219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.515185 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.515187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.515819 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.515821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.516307 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.516308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.516368 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.516778 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.516782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.517212 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.518945 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.518946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.518947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.519185 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.519395 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.519395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.519396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.519465 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.519830 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.519832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.519882 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.519960 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.520084 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:20.541220 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40149
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:20.620972 12616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:20.621206 12616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:20.621248 12616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:20.622376 12616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:20.622448 12616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:20.622488 12616 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:34:20.622536 12616 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:20.622575 12616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:20.624060 12616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:20.624333 12616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:20.624560 12616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:20.629921 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.629897 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.629951 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:20.630084 12616 server_base.cc:1034] running on GCE node
I20260620 18:34:20.630280 12616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:20.630532 12616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:20.631697 12616 hybrid_clock.cc:648] HybridClock initialized: now 1781980460631682 us; error 29 us; skew 500 ppm
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:20.634181 12616 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:20.634589 12616 webserver.cc:460] Webserver started at http://127.10.109.66:39051/ using document root <none> and password file <none>
I20260620 18:34:20.634881 12616 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:20.634953 12616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:20.635107 12616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:20.636205 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "100a85d087cd4c9ab30de64cdb67c146"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "88d21c5ea5a905e759896207e0004749"
server_key_iv: "18a24483b38b475942a9eea7f46a4f75"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.636571 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "100a85d087cd4c9ab30de64cdb67c146"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "88d21c5ea5a905e759896207e0004749"
server_key_iv: "18a24483b38b475942a9eea7f46a4f75"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.637919 12616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:20.638659 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.638841 12616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:20.638940 12616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "100a85d087cd4c9ab30de64cdb67c146"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "88d21c5ea5a905e759896207e0004749"
server_key_iv: "18a24483b38b475942a9eea7f46a4f75"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.639040 12616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:20.646358 12616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:20.648535 12616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:20.648706 12616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:20.649156 12616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:20.649214 12616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.649246 12616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:20.649271 12616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.656330 12616 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:40547
I20260620 18:34:20.656446 12746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:40547 every 8 connection(s)
I20260620 18:34:20.656723 12616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:20.659152 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.649836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:35887 (local address 127.10.109.126:40149)
0620 18:34:20.649890 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.649893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.650086 (+ 193us) server_negotiation.cc:407] Connection header received
0620 18:34:20.650553 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.650555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.650567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.650633 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.651533 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.652362 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.653337 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.653544 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.653633 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.656274 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.656276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.656277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.656283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.657721 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.658127 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.658128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.658128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.658186 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.658436 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.658437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.658437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.658553 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.658648 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.658964 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.659040 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:34:20.659324 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.649821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.66:35887)
0620 18:34:20.649967 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.649998 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.650449 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.650728 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.650733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.651125 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.651418 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.651425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.652456 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.652457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.653235 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.653237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.653655 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.653656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.653775 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.654225 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.654232 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.656165 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.657821 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.657827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.657829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.658035 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.658254 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.658255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.658256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.658365 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.658653 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.658659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.658835 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.658941 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.659098 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260620 18:34:20.659821 12747 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40149
I20260620 18:34:20.659905 12747 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:20.660084 12747 heartbeater.cc:511] Master 127.10.109.126:40149 requested a full tablet report, sending...
I20260620 18:34:20.660446 12403 ts_manager.cc:194] Registered new tserver with Master: 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547)
I20260620 18:34:20.660983 12403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:35887
I20260620 18:34:20.665699 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.660882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:38145 (local address 127.10.109.126:40149)
0620 18:34:20.660920 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.660922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.660976 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:34:20.661069 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.661070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.661086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.661140 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.661496 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.661931 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.662573 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.662799 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.662862 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.663703 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.663704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.663704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.663709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.664803 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.665125 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.665126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.665126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.665172 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.665335 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.665336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.665336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.665455 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.665502 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.665597 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.665648 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:20.665766 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.660790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.66:38145)
0620 18:34:20.660919 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.660930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.660997 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.661146 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.661147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.661357 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.661415 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.661417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.662023 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.662024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.662499 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.662500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.662870 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.662870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.662924 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.663365 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.663370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.663628 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.664873 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.664874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.664874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.665034 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.665233 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.665234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.665234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.665281 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.665496 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.665496 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.665528 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.665570 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.665637 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 18:34:20.665863 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12616
I20260620 18:34:20.665946 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:20.666150 10677 external_mini_cluster.cc:1428] Setting key a2f836748f832fcd73a3482dca2a6d63
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:20.693001 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:40149
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:20.776973 12756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:20.777200 12756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:20.777241 12756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:20.778362 12756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:20.778430 12756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:20.778465 12756 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:34:20.778514 12756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:20.778544 12756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:20.780014 12756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:20.780285 12756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:20.780515 12756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:20.785390 12762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.785396 12761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:20.785413 12764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:20.785478 12756 server_base.cc:1034] running on GCE node
I20260620 18:34:20.785753 12756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:20.786010 12756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:20.787207 12756 hybrid_clock.cc:648] HybridClock initialized: now 1781980460787179 us; error 42 us; skew 500 ppm
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:20.789577 12756 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:20.789990 12756 webserver.cc:460] Webserver started at http://127.10.109.67:35483/ using document root <none> and password file <none>
I20260620 18:34:20.790232 12756 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:20.790304 12756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:20.790457 12756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:20.791556 12756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "40555eab05f94e348d23b18db8312f92"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "e5a5aa37726bfbb5b80040a4a0fa2863"
server_key_iv: "7f9265fd8b404446418780b9319d2489"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.791906 12756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "40555eab05f94e348d23b18db8312f92"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "e5a5aa37726bfbb5b80040a4a0fa2863"
server_key_iv: "7f9265fd8b404446418780b9319d2489"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.793232 12756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:20.793978 12771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.794216 12756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:20.794324 12756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "40555eab05f94e348d23b18db8312f92"
format_stamp: "Formatted at 2026-06-20 18:34:20 on dist-test-slave-3jql"
server_key: "e5a5aa37726bfbb5b80040a4a0fa2863"
server_key_iv: "7f9265fd8b404446418780b9319d2489"
server_key_version: "encryptionkey@0"
I20260620 18:34:20.794406 12756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:20.823693 12756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:20.825603 12756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:20.825768 12756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:20.826362 12756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:20.826426 12756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.826485 12756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:20.826519 12756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.833449 12756 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:42629
I20260620 18:34:20.833563 12886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:42629 every 8 connection(s)
I20260620 18:34:20.833834 12756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:20.836488 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.826846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39251 (local address 127.10.109.126:40149)
0620 18:34:20.826907 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.826909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.827096 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:34:20.827626 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.827628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.827641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.827698 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.828408 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.829187 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.830823 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.831029 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.831121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.833775 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.833776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.833777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.833784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.835226 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.835603 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.835605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.835605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.835665 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.835929 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.835930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.835930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.836051 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.836112 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.836363 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.836426 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:20.836638 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.826826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.67:39251)
0620 18:34:20.826981 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.827015 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.827518 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.827768 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.827771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.828119 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.828297 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.828301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.829278 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.829279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.830693 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.830696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.831142 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.831143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.831273 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.831761 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.831770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.833697 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.835301 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.835306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.835308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.835509 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.835758 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.835759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.835759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.835860 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.836131 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.836136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.836243 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.836332 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.836442 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 18:34:20.837450 12887 heartbeater.cc:347] Connected to a master server at 127.10.109.126:40149
I20260620 18:34:20.837571 12887 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:20.837740 12887 heartbeater.cc:511] Master 127.10.109.126:40149 requested a full tablet report, sending...
I20260620 18:34:20.838148 12403 ts_manager.cc:194] Registered new tserver with Master: 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:20.838289 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 12756
I20260620 18:34:20.838392 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:20.838581 10677 external_mini_cluster.cc:1428] Setting key cf8f801d5841d19f922a6a8e8ad00249
I20260620 18:34:20.838675 12403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:39251
I20260620 18:34:20.839486 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:20.845080 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.838436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:37071 (local address 127.10.109.126:40149)
0620 18:34:20.838480 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.838482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.838678 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:34:20.838702 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.838703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.838714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.838802 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.839270 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.839950 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.840716 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.841051 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.841119 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.842241 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.842242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.842243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.842249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.843982 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.844306 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.844307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.844308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.844368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.844553 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.844554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.844554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.844688 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.844765 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.844966 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.845025 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:20.845228 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.838473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.10.109.67:37071)
0620 18:34:20.838589 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.838598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.838648 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.838931 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.838932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.839137 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.839199 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.839201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.840067 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.840068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.840584 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.840586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.841154 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.841155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.841237 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.841717 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.841722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.842129 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.844068 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.844069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.844069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.844227 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.844446 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.844447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.844447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.844492 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.844777 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.844778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.844821 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.844916 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.845021 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.863708 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.856379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.10.109.126:40149)
0620 18:34:20.856441 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.856443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.856470 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:34:20.856550 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.856550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.856560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.856643 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.857181 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.857648 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.858402 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.858594 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.858690 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.861088 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.861089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.861089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.861096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.862502 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.862936 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.862937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.862938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.862995 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.863206 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.863207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.863208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.863315 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.863418 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.863527 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.863603 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:34:20.865450 12403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:20.867148 12403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:20.874524 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.10.109.65:37095)
0620 18:34:20.870831 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.870835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.870844 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:20.870881 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.870883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.870898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.871003 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.871130 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.872244 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.873538 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874052 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874163 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874229 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.874391 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.874433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":81,"threads_started":1}
I20260620 18:34:20.874511 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.0.0.1:53892)
0620 18:34:20.870241 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.870303 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.870374 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.871005 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.871006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.871009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.871068 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.871071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.872334 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.872334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.873469 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.873471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874166 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.874254 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874267 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.874364 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.874422 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":43,"thread_start_us":111,"threads_started":1}
I20260620 18:34:20.874716 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40547 (local address 127.0.0.1:35246)
0620 18:34:20.870716 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.870726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.870881 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.871136 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.871137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.871139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.871167 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.871168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.872041 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.872042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.873377 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.873380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874326 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.874388 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874391 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.874441 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.874486 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":68,"threads_started":1}
I20260620 18:34:20.874893 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.10.109.66:40547)
0620 18:34:20.870705 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.870709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.870845 (+ 136us) server_negotiation.cc:407] Connection header received
0620 18:34:20.870979 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.870982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.870997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.871064 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.871252 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.871945 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.873467 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874197 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874254 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874300 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.874563 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.874804 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260620 18:34:20.874984 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42629 (local address 127.0.0.1:53782)
0620 18:34:20.870668 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.870680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.870757 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.870933 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.870936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.870939 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.871010 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.871012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.872310 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.872311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.873301 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.873304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874295 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.874412 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874415 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.874461 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.874490 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260620 18:34:20.875305 12541 tablet_service.cc:1467] Processing CreateTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (DEFAULT_TABLE table=test-table [id=7433670b89cb4e17bf8537268976797e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:20.875428 12681 tablet_service.cc:1467] Processing CreateTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (DEFAULT_TABLE table=test-table [id=7433670b89cb4e17bf8537268976797e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:20.875732 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ab2b1dedb46a39de2d6a112c4c1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.875759 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.870490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.10.109.67:42629)
0620 18:34:20.870672 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.870677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.870728 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:34:20.870826 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.870828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.870845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.870924 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.871093 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.872216 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.873403 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.874133 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.874204 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.874260 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.874530 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.875636 (+ 1106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260620 18:34:20.875767 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ab2b1dedb46a39de2d6a112c4c1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.876329 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (DEFAULT_TABLE table=test-table [id=7433670b89cb4e17bf8537268976797e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:20.876622 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ab2b1dedb46a39de2d6a112c4c1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.879806 12910 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Bootstrap starting.
I20260620 18:34:20.880637 12912 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Bootstrap starting.
I20260620 18:34:20.880736 12911 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Bootstrap starting.
I20260620 18:34:20.880779 12910 tablet_bootstrap.cc:654] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.881112 12910 log.cc:824] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:20.881314 12912 tablet_bootstrap.cc:654] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.881379 12911 tablet_bootstrap.cc:654] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.881649 12912 log.cc:824] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:20.881749 12911 log.cc:824] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:20.881913 12910 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: No bootstrap required, opened a new log
I20260620 18:34:20.881987 12910 ts_tablet_manager.cc:1397] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:20.882356 12912 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: No bootstrap required, opened a new log
I20260620 18:34:20.882429 12912 ts_tablet_manager.cc:1397] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:20.882581 12911 tablet_bootstrap.cc:492] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: No bootstrap required, opened a new log
I20260620 18:34:20.882647 12911 ts_tablet_manager.cc:1397] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:20.884009 12911 raft_consensus.cc:348] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884178 12911 raft_consensus.cc:374] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.884223 12911 raft_consensus.cc:729] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.884248 12910 raft_consensus.cc:348] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884407 12910 raft_consensus.cc:374] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.884361 12911 consensus_queue.cc:260] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884441 12910 raft_consensus.cc:729] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40555eab05f94e348d23b18db8312f92, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.884370 12912 raft_consensus.cc:348] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884502 12912 raft_consensus.cc:374] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.884544 12912 raft_consensus.cc:729] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.884570 12910 consensus_queue.cc:260] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884653 12911 ts_tablet_manager.cc:1428] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:34:20.884635 12912 consensus_queue.cc:260] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.884783 12607 heartbeater.cc:503] Master 127.10.109.126:40149 was elected leader, sending a full tablet report...
I20260620 18:34:20.884845 12912 ts_tablet_manager.cc:1428] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:34:20.884845 12910 ts_tablet_manager.cc:1428] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:34:20.884949 12887 heartbeater.cc:503] Master 127.10.109.126:40149 was elected leader, sending a full tablet report...
I20260620 18:34:20.884976 12747 heartbeater.cc:503] Master 127.10.109.126:40149 was elected leader, sending a full tablet report...
I20260620 18:34:20.893961 12918 raft_consensus.cc:482] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:20.894048 12918 raft_consensus.cc:504] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.894444 12918 leader_election.cc:288] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:20.897536 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.894787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:53391 (local address 127.10.109.65:37095)
0620 18:34:20.894842 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.894844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.894951 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:34:20.895027 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.895028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.895037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.895098 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.895254 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.895735 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.896788 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.897161 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.897275 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.897284 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.897435 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.897469 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:20.897544 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.894672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.10.109.66:53391)
0620 18:34:20.894876 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.894890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.894965 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.895102 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.895103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.895104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.895183 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.895185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.895833 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.895834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.896690 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.896693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.897272 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.897273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.897347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.897351 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.897427 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.897466 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260620 18:34:20.897630 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.894719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:51433 (local address 127.10.109.67:42629)
0620 18:34:20.894778 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.894780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.894789 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:20.894855 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.894856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.894867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.894981 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.895133 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.895674 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.896766 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.897244 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.897340 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.897348 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.897536 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.897572 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:34:20.897634 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.894650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42629 (local address 127.10.109.66:51433)
0620 18:34:20.894701 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.894713 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.894785 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.894988 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.894989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.894990 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.895071 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.895073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.895765 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.895766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.896672 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.896675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.897341 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.897342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.897446 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.897449 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.897528 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.897568 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":16}
I20260620 18:34:20.898023 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ab2b1dedb46a39de2d6a112c4c1e0" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92" is_pre_election: true
I20260620 18:34:20.898043 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ab2b1dedb46a39de2d6a112c4c1e0" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009" is_pre_election: true
I20260620 18:34:20.898159 12561 raft_consensus.cc:2454] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:20.898159 12841 raft_consensus.cc:2454] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:20.898416 12632 leader_election.cc:302] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 40555eab05f94e348d23b18db8312f92; no voters:
I20260620 18:34:20.898543 12918 raft_consensus.cc:2781] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:20.898607 12918 raft_consensus.cc:482] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:20.898661 12918 raft_consensus.cc:3037] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.899380 12918 raft_consensus.cc:504] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.899542 12918 leader_election.cc:288] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Requested vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:20.899639 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ab2b1dedb46a39de2d6a112c4c1e0" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009"
I20260620 18:34:20.899672 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ab2b1dedb46a39de2d6a112c4c1e0" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92"
I20260620 18:34:20.899718 12561 raft_consensus.cc:3037] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.899729 12841 raft_consensus.cc:3037] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.900558 12841 raft_consensus.cc:2454] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:20.900558 12561 raft_consensus.cc:2454] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:20.900724 12632 leader_election.cc:302] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 40555eab05f94e348d23b18db8312f92; no voters:
I20260620 18:34:20.900827 12918 raft_consensus.cc:2781] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:20.900965 12918 raft_consensus.cc:686] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Becoming Leader. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Running, Role: LEADER
I20260620 18:34:20.901031 12918 consensus_queue.cc:237] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.901798 12402 catalog_manager.cc:5495] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "100a85d087cd4c9ab30de64cdb67c146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } health_report { overall_health: UNKNOWN } } }
W20260620 18:34:20.907245 12748 tablet.cc:2369] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:34:20 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980460, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:20.931278 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.923533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.10.109.126:40149)
0620 18:34:20.923629 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.923632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.923742 (+ 110us) server_negotiation.cc:407] Connection header received
0620 18:34:20.923836 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.923837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.923848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.923919 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:20.924379 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.924856 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.925572 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.925746 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.925853 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.928261 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:20.928268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:20.928268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:20.928277 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:20.930012 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.930443 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.930444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.930445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.930506 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:20.930711 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:20.930712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:20.930712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:20.930855 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:20.930937 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.931093 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.931176 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:34:20.931387 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.923425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40149 (local address 127.0.0.1:60788)
0620 18:34:20.923655 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.923673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.923762 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.923936 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.923937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.924244 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:20.924291 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.924293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.924940 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.924941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.925449 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.925450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.925852 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.925853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.925908 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.926446 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:20.926451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:20.928065 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:20.930135 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.930143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.930145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.930347 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.930577 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:20.930578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:20.930579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:20.930638 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:20.930927 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:20.930933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:20.930977 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.931072 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.931259 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":85,"thread_start_us":111,"threads_started":1}
I20260620 18:34:20.933473 12402 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:60788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:34:20.933758 12402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:20.935361 12541 tablet_service.cc:1467] Processing CreateTablet for tablet e10fc19279764b568e585f02c20f7617 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d359d995a59042f0abf4fd95997346b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:20.935365 12821 tablet_service.cc:1467] Processing CreateTablet for tablet e10fc19279764b568e585f02c20f7617 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d359d995a59042f0abf4fd95997346b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:20.935542 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10fc19279764b568e585f02c20f7617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.935542 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10fc19279764b568e585f02c20f7617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.935521 12681 tablet_service.cc:1467] Processing CreateTablet for tablet e10fc19279764b568e585f02c20f7617 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d359d995a59042f0abf4fd95997346b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:20.935658 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10fc19279764b568e585f02c20f7617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:20.936822 12911 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: Bootstrap starting.
I20260620 18:34:20.937179 12910 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: Bootstrap starting.
I20260620 18:34:20.937178 12912 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: Bootstrap starting.
I20260620 18:34:20.937505 12911 tablet_bootstrap.cc:654] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.937638 12912 tablet_bootstrap.cc:654] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.937732 12910 tablet_bootstrap.cc:654] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:20.938139 12911 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: No bootstrap required, opened a new log
I20260620 18:34:20.938189 12911 ts_tablet_manager.cc:1397] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:20.938381 12912 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: No bootstrap required, opened a new log
I20260620 18:34:20.938345 12911 raft_consensus.cc:348] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.938414 12911 raft_consensus.cc:374] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.938421 12912 ts_tablet_manager.cc:1397] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:20.938438 12911 raft_consensus.cc:729] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.938486 12911 consensus_queue.cc:260] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.938598 12911 ts_tablet_manager.cc:1428] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.938591 12910 tablet_bootstrap.cc:492] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: No bootstrap required, opened a new log
I20260620 18:34:20.938567 12912 raft_consensus.cc:348] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.938642 12912 raft_consensus.cc:374] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.938643 12910 ts_tablet_manager.cc:1397] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:20.938668 12912 raft_consensus.cc:729] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.938714 12912 consensus_queue.cc:260] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.938719 12916 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.938817 12916 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.938834 12912 ts_tablet_manager.cc:1428] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.938807 12910 raft_consensus.cc:348] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.938879 12910 raft_consensus.cc:374] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:20.938912 12910 raft_consensus.cc:729] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40555eab05f94e348d23b18db8312f92, State: Initialized, Role: FOLLOWER
I20260620 18:34:20.938951 12910 consensus_queue.cc:260] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.939038 12910 ts_tablet_manager.cc:1428] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:20.939153 12917 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.939242 12917 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.939453 12918 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.939551 12918 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.965253 12916 raft_consensus.cc:482] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:20.965350 12916 raft_consensus.cc:504] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.965744 12916 leader_election.cc:288] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:20.969429 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.966005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:40571 (local address 127.10.109.67:42629)
0620 18:34:20.966045 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.966047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.966247 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:34:20.966273 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.966274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.966281 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.966357 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.966502 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.967229 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.968648 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.969021 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.969112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.969119 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.969316 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.969366 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:20.969435 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.965893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42629 (local address 127.10.109.65:40571)
0620 18:34:20.966152 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.966162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.966216 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.966380 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.966380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.966382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.966434 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.966435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.967308 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.967309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.968549 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.968552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.969135 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.969136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.969218 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.969222 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.969278 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.969310 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":34,"thread_start_us":116,"threads_started":1}
I20260620 18:34:20.969506 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.965964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:45037 (local address 127.10.109.66:40547)
0620 18:34:20.966011 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.966014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.966022 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:20.966048 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.966049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.966097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.966162 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:20.966342 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.967167 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.968670 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.969050 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.969196 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.969206 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.969392 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.969433 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:20.969696 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.965875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40547 (local address 127.10.109.65:45037)
0620 18:34:20.965930 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.965942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.965981 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.966172 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.966173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.966176 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:20.966257 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.966259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.967248 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.967249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.968549 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.968552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.969199 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.969201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.969300 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.969304 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.969361 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.969398 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":58}
I20260620 18:34:20.969751 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10fc19279764b568e585f02c20f7617" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92" is_pre_election: true
I20260620 18:34:20.969867 12841 raft_consensus.cc:2454] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 0.
I20260620 18:34:20.969993 12701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10fc19279764b568e585f02c20f7617" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100a85d087cd4c9ab30de64cdb67c146" is_pre_election: true
I20260620 18:34:20.970039 12492 leader_election.cc:302] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40555eab05f94e348d23b18db8312f92, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:20.970134 12701 raft_consensus.cc:2454] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 0.
I20260620 18:34:20.970165 12916 raft_consensus.cc:2781] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:20.970201 12916 raft_consensus.cc:482] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:20.970233 12916 raft_consensus.cc:3037] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.971033 12916 raft_consensus.cc:504] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.971210 12916 leader_election.cc:288] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 election: Requested vote from peers 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:20.971367 12701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10fc19279764b568e585f02c20f7617" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100a85d087cd4c9ab30de64cdb67c146"
I20260620 18:34:20.971359 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10fc19279764b568e585f02c20f7617" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92"
I20260620 18:34:20.971447 12701 raft_consensus.cc:3037] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.971446 12841 raft_consensus.cc:3037] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:20.972275 12841 raft_consensus.cc:2454] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 1.
I20260620 18:34:20.972275 12701 raft_consensus.cc:2454] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 1.
I20260620 18:34:20.972442 12492 leader_election.cc:302] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40555eab05f94e348d23b18db8312f92, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:20.972560 12916 raft_consensus.cc:2781] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:20.972681 12916 raft_consensus.cc:686] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [term 1 LEADER]: Becoming Leader. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Running, Role: LEADER
I20260620 18:34:20.972796 12916 consensus_queue.cc:237] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:20.973243 12932 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: TxnStatusTablet state changed. Reason: New leader 4b49a46884144c94973980f1b8a39009. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.973330 12932 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:20.973506 12934 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:34:20.973632 12934 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:34:20.973681 12402 catalog_manager.cc:5495] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b49a46884144c94973980f1b8a39009 (127.10.109.65). New cstate: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:20.989950 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.986615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.0.0.1:53896)
0620 18:34:20.986734 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:20.986796 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:20.987024 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:20.987241 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:20.987241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:20.987243 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:20.987305 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.987309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.988287 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.988288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.988855 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:20.988856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.989403 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.989404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:20.989465 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.989479 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:20.989774 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:20.989780 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:34:20.989825 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 18:34:20.989863 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:34:20.989946 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.986700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.10.109.65:37095)
0620 18:34:20.986759 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.986762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.986984 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:34:20.987124 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.987126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.987145 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.987229 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:20.987379 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.988196 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.988978 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.989317 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.989395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.989577 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:20.989701 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:20.989788 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.989829 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 18:34:20.989850 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:20.991865 12841 raft_consensus.cc:1261] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Refusing update from remote peer 4b49a46884144c94973980f1b8a39009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:20.991873 12701 raft_consensus.cc:1261] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Refusing update from remote peer 4b49a46884144c94973980f1b8a39009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:20.992194 12916 consensus_queue.cc:1035] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:20.992321 12932 consensus_queue.cc:1035] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:20.993448 12918 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: TxnStatusTablet state changed. Reason: New leader 4b49a46884144c94973980f1b8a39009. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.993543 12918 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.993533 12917 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: TxnStatusTablet state changed. Reason: New leader 4b49a46884144c94973980f1b8a39009. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.993615 12917 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.994925 12916 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.995019 12916 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:20.995069 12916 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.995069 12918 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.995119 12916 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 4b49a46884144c94973980f1b8a39009: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:20.995137 12918 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 100a85d087cd4c9ab30de64cdb67c146: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.995188 12937 mvcc.cc:205] Tried to move back new op lower bound from 7298991968220340224 to 7298991968145088512. Current Snapshot: MvccSnapshot[applied={T|T < 7298991968220340224}]
I20260620 18:34:20.995396 12917 tablet_replica.cc:439] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } } }
I20260620 18:34:20.995476 12917 tablet_replica.cc:442] T e10fc19279764b568e585f02c20f7617 P 40555eab05f94e348d23b18db8312f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:20.996562 12938 mvcc.cc:205] Tried to move back new op lower bound from 7298991968220340224 to 7298991968145088512. Current Snapshot: MvccSnapshot[applied={T|T < 7298991968220340224}]
I20260620 18:34:20.996656 12939 mvcc.cc:205] Tried to move back new op lower bound from 7298991968220340224 to 7298991968145088512. Current Snapshot: MvccSnapshot[applied={T|T < 7298991968220340224 or (T in {7298991968220340224})}]
I20260620 18:34:21.000085 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:20.997619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.10.109.66:40547)
0620 18:34:20.997662 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:20.997664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:20.997673 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:20.997748 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:20.997749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:20.997764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:20.997829 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:20.997982 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.998457 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.999310 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:20.999512 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:20.999606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:20.999777 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:20.999876 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:20.999967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:20.999990 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 18:34:21.000008 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:21.005084 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:21.002500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:33489 (local address 127.10.109.65:37095)
0620 18:34:21.002557 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:21.002559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:21.002568 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:21.002624 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:21.002625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:21.002635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:21.002690 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:21.002874 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.003595 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.004336 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.004632 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.004702 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:21.004853 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:21.004883 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:21.004939 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:21.004999 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 18:34:21.005016 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:21.005061 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:21.002421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.10.109.66:33489)
0620 18:34:21.002489 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:21.002501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:21.002571 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:21.002707 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:21.002708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:21.002709 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:21.002798 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:21.002799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.003678 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.003679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:21.004250 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:21.004252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.004706 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.004707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:21.004754 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:21.004766 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:21.004949 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:21.004951 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:21.004982 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 18:34:21.005014 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 18:34:21.007748 12841 raft_consensus.cc:1261] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:21.007784 12561 raft_consensus.cc:1261] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:21.007999 12920 consensus_queue.cc:1035] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:21.008111 12918 consensus_queue.cc:1035] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:34:21.009546 12608 tablet.cc:2369] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:21.016784 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:21.014003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40547 (local address 127.10.109.65:60873)
0620 18:34:21.014084 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:21.014095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:21.014138 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:21.014373 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:21.014373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:21.014375 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:21.014431 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:21.014432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.015095 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.015096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:21.016031 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:21.016034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.016379 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.016380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:21.016457 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:21.016473 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:21.016666 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:21.016668 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:21.016704 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 18:34:21.016737 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:34:21.016796 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:21.014206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:60873 (local address 127.10.109.66:40547)
0620 18:34:21.014270 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:21.014271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:21.014278 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:34:21.014296 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:21.014296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:21.014303 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:21.014347 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:21.014494 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.015024 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.016121 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:21.016292 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:21.016380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:21.016561 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:21.016596 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:21.016653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:21.016707 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:34:21.016728 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:34:21.026914 12402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:21.027012 12402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:21.027697 12402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Sending DeleteTablet for 3 replicas of tablet 950ab2b1dedb46a39de2d6a112c4c1e0
I20260620 18:34:21.028025 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:21 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:35246
I20260620 18:34:21.028054 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:21 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53782
I20260620 18:34:21.028205 12541 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ab2b1dedb46a39de2d6a112c4c1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:21 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53892
I20260620 18:34:21.028374 12955 tablet_replica.cc:330] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: stopping tablet replica
I20260620 18:34:21.028383 12402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:34:21.028462 12955 raft_consensus.cc:2229] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Raft consensus shutting down.
W20260620 18:34:21.028506 12402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:21.028545 12955 raft_consensus.cc:2258] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:21.028955 12955 ts_tablet_manager.cc:1905] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:21.029824 12541 tablet_service.cc:1467] Processing CreateTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 (DEFAULT_TABLE table=test-table [id=57e538e08d79497d80ccdb88428d46e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:21.029937 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6cb2ad57fb41e786cd7c5df418cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:21.030668 12955 ts_tablet_manager.cc:1918] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:21.030727 12955 log.cc:1196] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/950ab2b1dedb46a39de2d6a112c4c1e0
I20260620 18:34:21.031009 12955 ts_tablet_manager.cc:1939] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 100a85d087cd4c9ab30de64cdb67c146: Deleting consensus metadata
I20260620 18:34:21.031306 12911 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Bootstrap starting.
I20260620 18:34:21.031353 12681 tablet_service.cc:1467] Processing CreateTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 (DEFAULT_TABLE table=test-table [id=57e538e08d79497d80ccdb88428d46e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:21.031450 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6cb2ad57fb41e786cd7c5df418cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:21.031688 12911 tablet_bootstrap.cc:654] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:21.032162 12391 catalog_manager.cc:4841] TS 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547): tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (table test-table [id=7433670b89cb4e17bf8537268976797e]) successfully deleted
I20260620 18:34:21.032558 12912 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Bootstrap starting.
I20260620 18:34:21.032569 12911 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: No bootstrap required, opened a new log
I20260620 18:34:21.032609 12911 ts_tablet_manager.cc:1397] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:21.032737 12911 raft_consensus.cc:348] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.032792 12911 raft_consensus.cc:374] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:21.032812 12911 raft_consensus.cc:729] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Initialized, Role: FOLLOWER
I20260620 18:34:21.032855 12911 consensus_queue.cc:260] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.032920 12912 tablet_bootstrap.cc:654] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:21.032963 12911 ts_tablet_manager.cc:1428] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:21.033492 12912 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: No bootstrap required, opened a new log
I20260620 18:34:21.033531 12912 ts_tablet_manager.cc:1397] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:21.033612 12821 tablet_service.cc:1467] Processing CreateTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 (DEFAULT_TABLE table=test-table [id=57e538e08d79497d80ccdb88428d46e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:21.033651 12912 raft_consensus.cc:348] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.033712 12912 raft_consensus.cc:374] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:21.033720 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6cb2ad57fb41e786cd7c5df418cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:21.033733 12912 raft_consensus.cc:729] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Initialized, Role: FOLLOWER
I20260620 18:34:21.033772 12912 consensus_queue.cc:260] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.033843 12912 ts_tablet_manager.cc:1428] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:21.035321 12910 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Bootstrap starting.
I20260620 18:34:21.035696 12910 tablet_bootstrap.cc:654] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:21.036341 12910 tablet_bootstrap.cc:492] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: No bootstrap required, opened a new log
I20260620 18:34:21.036388 12910 ts_tablet_manager.cc:1397] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:21.036518 12910 raft_consensus.cc:348] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.036582 12910 raft_consensus.cc:374] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:21.036605 12910 raft_consensus.cc:729] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40555eab05f94e348d23b18db8312f92, State: Initialized, Role: FOLLOWER
I20260620 18:34:21.036645 12910 consensus_queue.cc:260] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.036712 12910 ts_tablet_manager.cc:1428] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:21.037313 12956 tablet_replica.cc:330] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: stopping tablet replica
I20260620 18:34:21.037469 12957 tablet_replica.cc:330] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: stopping tablet replica
I20260620 18:34:21.037609 12957 raft_consensus.cc:2229] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:21.037734 12956 raft_consensus.cc:2229] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:21.037827 12957 raft_consensus.cc:2258] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:21.037912 12956 raft_consensus.cc:2258] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:21.038411 12956 ts_tablet_manager.cc:1905] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:21.038764 12957 ts_tablet_manager.cc:1905] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:21.040306 12956 ts_tablet_manager.cc:1918] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:21.040380 12956 log.cc:1196] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/950ab2b1dedb46a39de2d6a112c4c1e0
I20260620 18:34:21.040630 12956 ts_tablet_manager.cc:1939] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 40555eab05f94e348d23b18db8312f92: Deleting consensus metadata
I20260620 18:34:21.040855 12957 ts_tablet_manager.cc:1918] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:21.041062 12388 catalog_manager.cc:4841] TS 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629): tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (table test-table [id=7433670b89cb4e17bf8537268976797e]) successfully deleted
I20260620 18:34:21.041191 12957 log.cc:1196] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/950ab2b1dedb46a39de2d6a112c4c1e0
I20260620 18:34:21.041862 12957 ts_tablet_manager.cc:1939] T 950ab2b1dedb46a39de2d6a112c4c1e0 P 4b49a46884144c94973980f1b8a39009: Deleting consensus metadata
I20260620 18:34:21.042290 12391 catalog_manager.cc:4841] TS 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095): tablet 950ab2b1dedb46a39de2d6a112c4c1e0 (table test-table [id=7433670b89cb4e17bf8537268976797e]) successfully deleted
I20260620 18:34:21.066944 12920 raft_consensus.cc:482] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:21.067049 12920 raft_consensus.cc:504] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.067230 12920 leader_election.cc:288] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:21.067459 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6cb2ad57fb41e786cd7c5df418cf29" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92" is_pre_election: true
I20260620 18:34:21.067463 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6cb2ad57fb41e786cd7c5df418cf29" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009" is_pre_election: true
I20260620 18:34:21.067564 12841 raft_consensus.cc:2454] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:21.067564 12561 raft_consensus.cc:2454] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:21.067703 12632 leader_election.cc:302] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 40555eab05f94e348d23b18db8312f92; no voters:
I20260620 18:34:21.067803 12920 raft_consensus.cc:2781] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:21.067834 12920 raft_consensus.cc:482] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:21.067854 12920 raft_consensus.cc:3037] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:21.068720 12920 raft_consensus.cc:504] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.068849 12920 leader_election.cc:288] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Requested vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:21.069023 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6cb2ad57fb41e786cd7c5df418cf29" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009"
I20260620 18:34:21.069096 12561 raft_consensus.cc:3037] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:21.069104 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6cb2ad57fb41e786cd7c5df418cf29" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92"
I20260620 18:34:21.069171 12841 raft_consensus.cc:3037] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:21.069880 12841 raft_consensus.cc:2454] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:21.069880 12561 raft_consensus.cc:2454] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:21.070088 12635 leader_election.cc:302] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:21.070250 12920 raft_consensus.cc:2781] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:21.070299 12920 raft_consensus.cc:686] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Becoming Leader. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Running, Role: LEADER
I20260620 18:34:21.070343 12920 consensus_queue.cc:237] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:21.070895 12402 catalog_manager.cc:5495] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "100a85d087cd4c9ab30de64cdb67c146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:21.079228 12561 raft_consensus.cc:1261] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:21.079383 12841 raft_consensus.cc:1261] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:21.079536 12951 consensus_queue.cc:1035] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:21.079737 12951 consensus_queue.cc:1035] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:34:21.085592 12939 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:34:22.607625 12402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:22.607745 12402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:22.608717 12402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Sending DeleteTablet for 3 replicas of tablet ae6cb2ad57fb41e786cd7c5df418cf29
I20260620 18:34:22.609146 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:35246
I20260620 18:34:22.609185 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53782
I20260620 18:34:22.609344 12541 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6cb2ad57fb41e786cd7c5df418cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53892
I20260620 18:34:22.609349 12985 tablet_replica.cc:330] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: stopping tablet replica
I20260620 18:34:22.609421 12985 raft_consensus.cc:2229] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:22.609484 12985 raft_consensus.cc:2258] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.609741 12985 ts_tablet_manager.cc:1905] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.611562 12985 ts_tablet_manager.cc:1918] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:34:22.611606 12985 log.cc:1196] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/ae6cb2ad57fb41e786cd7c5df418cf29
I20260620 18:34:22.611794 12985 ts_tablet_manager.cc:1939] T ae6cb2ad57fb41e786cd7c5df418cf29 P 100a85d087cd4c9ab30de64cdb67c146: Deleting consensus metadata
I20260620 18:34:22.611827 12986 tablet_replica.cc:330] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: stopping tablet replica
I20260620 18:34:22.611882 12986 raft_consensus.cc:2229] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:22.611913 12986 raft_consensus.cc:2258] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.612098 12987 tablet_replica.cc:330] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: stopping tablet replica
I20260620 18:34:22.612138 12986 ts_tablet_manager.cc:1905] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.612183 12987 raft_consensus.cc:2229] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:22.612228 12987 raft_consensus.cc:2258] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.612519 12391 catalog_manager.cc:4841] TS 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547): tablet ae6cb2ad57fb41e786cd7c5df418cf29 (table test-table [id=57e538e08d79497d80ccdb88428d46e3]) successfully deleted
I20260620 18:34:22.612597 12987 ts_tablet_manager.cc:1905] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.613940 12986 ts_tablet_manager.cc:1918] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:34:22.614010 12986 log.cc:1196] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/ae6cb2ad57fb41e786cd7c5df418cf29
I20260620 18:34:22.614190 12987 ts_tablet_manager.cc:1918] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:34:22.614253 12987 log.cc:1196] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/ae6cb2ad57fb41e786cd7c5df418cf29
I20260620 18:34:22.614287 12986 ts_tablet_manager.cc:1939] T ae6cb2ad57fb41e786cd7c5df418cf29 P 40555eab05f94e348d23b18db8312f92: Deleting consensus metadata
I20260620 18:34:22.614455 12987 ts_tablet_manager.cc:1939] T ae6cb2ad57fb41e786cd7c5df418cf29 P 4b49a46884144c94973980f1b8a39009: Deleting consensus metadata
I20260620 18:34:22.614614 12388 catalog_manager.cc:4841] TS 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629): tablet ae6cb2ad57fb41e786cd7c5df418cf29 (table test-table [id=57e538e08d79497d80ccdb88428d46e3]) successfully deleted
I20260620 18:34:22.614902 12391 catalog_manager.cc:4841] TS 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095): tablet ae6cb2ad57fb41e786cd7c5df418cf29 (table test-table [id=57e538e08d79497d80ccdb88428d46e3]) successfully deleted
Jun 20 18:34:22 dist-test-slave-3jql krb5kdc[12359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:34:22 dist-test-slave-3jql krb5kdc[12359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:22.636897 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:22.628451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.10.109.126:40149)
0620 18:34:22.628740 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:22.628741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:22.628750 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:22.628905 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:22.628908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:22.628927 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:22.629047 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:22.629705 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.630231 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.631116 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.631319 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.631389 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:22.633943 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:22.633946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:22.633946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:22.633956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:22.635744 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.636051 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.636052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.636052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.636142 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:22.636349 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:22.636350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:22.636350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:22.636450 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:22.636522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:22.636746 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 18:34:22.636830 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":227,"threads_started":1}
I20260620 18:34:22.638549 12402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:22.638684 12402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:22.640218 12821 tablet_service.cc:1467] Processing CreateTablet for tablet e141c837506344e494feb6dcbdf615dd (DEFAULT_TABLE table=test-table [id=2b11c16da28847d3a028c96cb27a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.640218 12541 tablet_service.cc:1467] Processing CreateTablet for tablet e141c837506344e494feb6dcbdf615dd (DEFAULT_TABLE table=test-table [id=2b11c16da28847d3a028c96cb27a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.640345 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141c837506344e494feb6dcbdf615dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.640345 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141c837506344e494feb6dcbdf615dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.641723 12996 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Bootstrap starting.
I20260620 18:34:22.641723 12681 tablet_service.cc:1467] Processing CreateTablet for tablet e141c837506344e494feb6dcbdf615dd (DEFAULT_TABLE table=test-table [id=2b11c16da28847d3a028c96cb27a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.641832 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141c837506344e494feb6dcbdf615dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.642194 12996 tablet_bootstrap.cc:654] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.642576 12997 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Bootstrap starting.
I20260620 18:34:22.642942 12996 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: No bootstrap required, opened a new log
I20260620 18:34:22.642988 12997 tablet_bootstrap.cc:654] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.643054 12996 ts_tablet_manager.cc:1397] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:22.643209 12996 raft_consensus.cc:348] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.643342 12999 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Bootstrap starting.
I20260620 18:34:22.643384 12996 raft_consensus.cc:374] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.643442 12996 raft_consensus.cc:729] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40555eab05f94e348d23b18db8312f92, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.643522 12996 consensus_queue.cc:260] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.643713 12997 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: No bootstrap required, opened a new log
I20260620 18:34:22.643733 12999 tablet_bootstrap.cc:654] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.643755 12996 ts_tablet_manager.cc:1428] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:22.643841 12997 ts_tablet_manager.cc:1397] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:22.644002 12997 raft_consensus.cc:348] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.644105 12997 raft_consensus.cc:374] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.644141 12997 raft_consensus.cc:729] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.644194 12997 consensus_queue.cc:260] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.644310 12997 ts_tablet_manager.cc:1428] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:22.644718 12999 tablet_bootstrap.cc:492] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: No bootstrap required, opened a new log
I20260620 18:34:22.644786 12999 ts_tablet_manager.cc:1397] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:22.644892 12999 raft_consensus.cc:348] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.644974 12999 raft_consensus.cc:374] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.645021 12999 raft_consensus.cc:729] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.645080 12999 consensus_queue.cc:260] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.645192 12999 ts_tablet_manager.cc:1428] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:22.772783 12979 raft_consensus.cc:482] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:22.772890 12979 raft_consensus.cc:504] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.773176 12979 leader_election.cc:288] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:22.773401 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141c837506344e494feb6dcbdf615dd" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009" is_pre_election: true
I20260620 18:34:22.773540 12561 raft_consensus.cc:2454] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:22.773533 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141c837506344e494feb6dcbdf615dd" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92" is_pre_election: true
I20260620 18:34:22.773640 12841 raft_consensus.cc:2454] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 0.
I20260620 18:34:22.773806 12635 leader_election.cc:302] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:22.773926 12979 raft_consensus.cc:2781] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:22.773963 12979 raft_consensus.cc:482] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:22.773983 12979 raft_consensus.cc:3037] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:22.774689 12979 raft_consensus.cc:504] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.774925 12979 leader_election.cc:288] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Requested vote from peers 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:22.775097 12561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141c837506344e494feb6dcbdf615dd" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b49a46884144c94973980f1b8a39009"
I20260620 18:34:22.775132 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141c837506344e494feb6dcbdf615dd" candidate_uuid: "100a85d087cd4c9ab30de64cdb67c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92"
I20260620 18:34:22.775202 12841 raft_consensus.cc:3037] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:22.775205 12561 raft_consensus.cc:3037] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:22.775817 12561 raft_consensus.cc:2454] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:22.775818 12841 raft_consensus.cc:2454] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a85d087cd4c9ab30de64cdb67c146 in term 1.
I20260620 18:34:22.776023 12635 leader_election.cc:302] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a85d087cd4c9ab30de64cdb67c146, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:22.776144 12979 raft_consensus.cc:2781] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:22.776216 12979 raft_consensus.cc:686] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Becoming Leader. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Running, Role: LEADER
I20260620 18:34:22.776274 12979 consensus_queue.cc:237] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.776985 12402 catalog_manager.cc:5495] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "100a85d087cd4c9ab30de64cdb67c146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:22.796909 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:22.793504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.10.109.66:40547)
0620 18:34:22.793689 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:22.793692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:22.793700 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:22.793722 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:22.793723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:22.793742 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:22.793809 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:22.794136 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.794709 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.795915 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:22.796115 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:22.796206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:22.796529 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:22.796569 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:22.796623 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:22.796826 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:34:22.796848 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":141,"threads_started":1}
I20260620 18:34:22.799782 12561 raft_consensus.cc:1261] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:22.799782 12841 raft_consensus.cc:1261] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Refusing update from remote peer 100a85d087cd4c9ab30de64cdb67c146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:22.800011 12979 consensus_queue.cc:1035] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:22.800118 12918 consensus_queue.cc:1035] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:22.810051 12402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:22.810202 12402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:22.811144 12402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Sending DeleteTablet for 3 replicas of tablet e141c837506344e494feb6dcbdf615dd
I20260620 18:34:22.811414 12541 tablet_service.cc:1514] Processing DeleteTablet for tablet e141c837506344e494feb6dcbdf615dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53892
I20260620 18:34:22.811431 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet e141c837506344e494feb6dcbdf615dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53782
I20260620 18:34:22.811522 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet e141c837506344e494feb6dcbdf615dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:22 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:35246
I20260620 18:34:22.811625 12987 tablet_replica.cc:330] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: stopping tablet replica
I20260620 18:34:22.811690 12987 raft_consensus.cc:2229] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:22.811734 12987 raft_consensus.cc:2258] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.811715 12402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:34:22.811802 12986 tablet_replica.cc:330] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: stopping tablet replica
I20260620 18:34:22.811849 12986 raft_consensus.cc:2229] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 18:34:22.811864 12402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:22.811902 12986 raft_consensus.cc:2258] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.811967 12985 tablet_replica.cc:330] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: stopping tablet replica
I20260620 18:34:22.812022 12985 raft_consensus.cc:2229] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:22.812089 12985 raft_consensus.cc:2258] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:22.812412 12987 ts_tablet_manager.cc:1905] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.812472 12985 ts_tablet_manager.cc:1905] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.812620 12986 ts_tablet_manager.cc:1905] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:22.814260 12987 ts_tablet_manager.cc:1918] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:22.814321 12987 log.cc:1196] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/e141c837506344e494feb6dcbdf615dd
I20260620 18:34:22.814353 12985 ts_tablet_manager.cc:1918] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:22.814419 12985 log.cc:1196] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/e141c837506344e494feb6dcbdf615dd
I20260620 18:34:22.814569 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 (DEFAULT_TABLE table=test-table [id=0caee372ced446bd839d92449fd8e75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.814553 12987 ts_tablet_manager.cc:1939] T e141c837506344e494feb6dcbdf615dd P 4b49a46884144c94973980f1b8a39009: Deleting consensus metadata
I20260620 18:34:22.814651 12985 ts_tablet_manager.cc:1939] T e141c837506344e494feb6dcbdf615dd P 100a85d087cd4c9ab30de64cdb67c146: Deleting consensus metadata
I20260620 18:34:22.814683 12681 tablet_service.cc:1467] Processing CreateTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 (DEFAULT_TABLE table=test-table [id=0caee372ced446bd839d92449fd8e75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.814554 12541 tablet_service.cc:1467] Processing CreateTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 (DEFAULT_TABLE table=test-table [id=0caee372ced446bd839d92449fd8e75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:22.814831 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355a2f350f24fe4b607e140dbe8b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.814841 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355a2f350f24fe4b607e140dbe8b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.815651 12997 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009: Bootstrap starting.
I20260620 18:34:22.815666 12999 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Bootstrap starting.
I20260620 18:34:22.815964 12999 tablet_bootstrap.cc:654] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.815964 12997 tablet_bootstrap.cc:654] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.816393 12986 ts_tablet_manager.cc:1918] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:22.816401 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355a2f350f24fe4b607e140dbe8b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:22.816468 12986 log.cc:1196] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/e141c837506344e494feb6dcbdf615dd
I20260620 18:34:22.816673 12986 ts_tablet_manager.cc:1939] T e141c837506344e494feb6dcbdf615dd P 40555eab05f94e348d23b18db8312f92: Deleting consensus metadata
I20260620 18:34:22.816987 12388 catalog_manager.cc:4841] TS 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629): tablet e141c837506344e494feb6dcbdf615dd (table test-table [id=2b11c16da28847d3a028c96cb27a7f64]) successfully deleted
I20260620 18:34:22.817035 12999 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: No bootstrap required, opened a new log
I20260620 18:34:22.817081 12999 ts_tablet_manager.cc:1397] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:22.817199 12999 raft_consensus.cc:348] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.817269 12999 raft_consensus.cc:374] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.817288 12999 raft_consensus.cc:729] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a85d087cd4c9ab30de64cdb67c146, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.817327 12999 consensus_queue.cc:260] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.817413 12999 ts_tablet_manager.cc:1428] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:22.817665 12996 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Bootstrap starting.
I20260620 18:34:22.817873 12997 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009: No bootstrap required, opened a new log
I20260620 18:34:22.818061 12996 tablet_bootstrap.cc:654] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:22.818111 12997 ts_tablet_manager.cc:1397] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:22.818320 12391 catalog_manager.cc:4841] TS 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547): tablet e141c837506344e494feb6dcbdf615dd (table test-table [id=2b11c16da28847d3a028c96cb27a7f64]) successfully deleted
I20260620 18:34:22.818442 12997 raft_consensus.cc:348] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.818555 12997 raft_consensus.cc:374] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.818585 12997 raft_consensus.cc:729] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.818637 12997 consensus_queue.cc:260] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.818675 12391 catalog_manager.cc:4841] TS 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095): tablet e141c837506344e494feb6dcbdf615dd (table test-table [id=2b11c16da28847d3a028c96cb27a7f64]) successfully deleted
I20260620 18:34:22.818794 12997 ts_tablet_manager.cc:1428] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:22.819157 12996 tablet_bootstrap.cc:492] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: No bootstrap required, opened a new log
I20260620 18:34:22.819233 12996 ts_tablet_manager.cc:1397] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:22.819378 12996 raft_consensus.cc:348] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.819479 12996 raft_consensus.cc:374] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:22.819507 12996 raft_consensus.cc:729] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40555eab05f94e348d23b18db8312f92, State: Initialized, Role: FOLLOWER
I20260620 18:34:22.819548 12996 consensus_queue.cc:260] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.819679 12996 ts_tablet_manager.cc:1428] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 18:34:22.836083 12888 tablet.cc:2369] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:22.997454 12964 raft_consensus.cc:482] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:22.997579 12964 raft_consensus.cc:504] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.997763 12964 leader_election.cc:288] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:22.998242 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355a2f350f24fe4b607e140dbe8b307" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92" is_pre_election: true
I20260620 18:34:22.998379 12841 raft_consensus.cc:2454] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 0.
I20260620 18:34:22.998350 12701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355a2f350f24fe4b607e140dbe8b307" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100a85d087cd4c9ab30de64cdb67c146" is_pre_election: true
I20260620 18:34:22.998575 12701 raft_consensus.cc:2454] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 0.
I20260620 18:34:22.998736 12492 leader_election.cc:302] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40555eab05f94e348d23b18db8312f92, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:22.998893 12964 raft_consensus.cc:2781] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:22.998943 12964 raft_consensus.cc:482] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:22.998963 12964 raft_consensus.cc:3037] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:22.999753 12964 raft_consensus.cc:504] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:22.999905 12964 leader_election.cc:288] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 election: Requested vote from peers 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547), 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629)
I20260620 18:34:23.000078 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355a2f350f24fe4b607e140dbe8b307" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40555eab05f94e348d23b18db8312f92"
I20260620 18:34:23.000097 12701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355a2f350f24fe4b607e140dbe8b307" candidate_uuid: "4b49a46884144c94973980f1b8a39009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100a85d087cd4c9ab30de64cdb67c146"
I20260620 18:34:23.000175 12841 raft_consensus.cc:3037] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:23.000187 12701 raft_consensus.cc:3037] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:23.000767 12841 raft_consensus.cc:2454] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 1.
I20260620 18:34:23.000841 12701 raft_consensus.cc:2454] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b49a46884144c94973980f1b8a39009 in term 1.
I20260620 18:34:23.000990 12492 leader_election.cc:302] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40555eab05f94e348d23b18db8312f92, 4b49a46884144c94973980f1b8a39009; no voters:
I20260620 18:34:23.001083 12964 raft_consensus.cc:2781] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:23.001147 12964 raft_consensus.cc:686] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [term 1 LEADER]: Becoming Leader. State: Replica: 4b49a46884144c94973980f1b8a39009, State: Running, Role: LEADER
I20260620 18:34:23.001199 12964 consensus_queue.cc:237] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } }
I20260620 18:34:23.001807 12402 catalog_manager.cc:5495] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b49a46884144c94973980f1b8a39009 (127.10.109.65). New cstate: current_term: 1 leader_uuid: "4b49a46884144c94973980f1b8a39009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b49a46884144c94973980f1b8a39009" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:23.054224 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.051682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.10.109.65:37095)
0620 18:34:23.051876 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.051878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.051896 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:34:23.051914 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.051914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.051923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.051999 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:23.052115 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.052707 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.053476 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.053694 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.053768 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.053922 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:23.053955 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:23.054010 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.054107 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.054131 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260620 18:34:23.059757 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.057123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:42157 (local address 127.10.109.65:37095)
0620 18:34:23.057177 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:23.057179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:23.057187 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:23.057238 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:23.057239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:23.057247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:23.057294 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:23.057390 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.058048 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.058867 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.059045 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.059150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.059425 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:23.059467 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:23.059529 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:23.059670 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:34:23.059702 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:34:23.059841 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:23.056911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.10.109.65:42157)
0620 18:34:23.057038 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:23.057048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:23.057173 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:23.057302 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:23.057303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:23.057305 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:23.057343 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.057344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.058206 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.058207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.058779 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:23.058781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:23.059219 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:23.059221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:23.059298 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:23.059306 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:23.059564 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:23.059567 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:23.059620 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:34:23.059678 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":86,"threads_started":1}
I20260620 18:34:23.062302 12841 raft_consensus.cc:1261] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Refusing update from remote peer 4b49a46884144c94973980f1b8a39009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:23.062546 12701 raft_consensus.cc:1261] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Refusing update from remote peer 4b49a46884144c94973980f1b8a39009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:23.062606 13004 consensus_queue.cc:1035] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40555eab05f94e348d23b18db8312f92" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:23.062865 13004 consensus_queue.cc:1035] T 2355a2f350f24fe4b607e140dbe8b307 P 4b49a46884144c94973980f1b8a39009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100a85d087cd4c9ab30de64cdb67c146" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:34:23.067277 12965 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:34:24.588275 12403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:24.588369 12403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:24.589185 12403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67575e68329429d86e1a823c80636e8: Sending DeleteTablet for 3 replicas of tablet 2355a2f350f24fe4b607e140dbe8b307
I20260620 18:34:24.589416 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:24 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53782
I20260620 18:34:24.589469 12541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:24 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53892
I20260620 18:34:24.589529 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12476
I20260620 18:34:24.589653 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet 2355a2f350f24fe4b607e140dbe8b307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:24 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:35246
I20260620 18:34:24.590633 13043 tablet_replica.cc:330] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: stopping tablet replica
I20260620 18:34:24.590705 13043 raft_consensus.cc:2229] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:24.590765 13043 raft_consensus.cc:2258] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:24.591250 13043 ts_tablet_manager.cc:1905] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:24.593374 13043 ts_tablet_manager.cc:1918] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:34:24.593444 13043 log.cc:1196] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/2355a2f350f24fe4b607e140dbe8b307
I20260620 18:34:24.593676 13043 ts_tablet_manager.cc:1939] T 2355a2f350f24fe4b607e140dbe8b307 P 100a85d087cd4c9ab30de64cdb67c146: Deleting consensus metadata
I20260620 18:34:24.594152 12391 catalog_manager.cc:4841] TS 100a85d087cd4c9ab30de64cdb67c146 (127.10.109.66:40547): tablet 2355a2f350f24fe4b607e140dbe8b307 (table test-table [id=0caee372ced446bd839d92449fd8e75b]) successfully deleted
I20260620 18:34:24.594857 13041 tablet_replica.cc:330] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: stopping tablet replica
I20260620 18:34:24.594940 13041 raft_consensus.cc:2229] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:24.594996 13041 raft_consensus.cc:2258] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:24.595532 13041 ts_tablet_manager.cc:1905] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:24.596827 13041 ts_tablet_manager.cc:1918] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:34:24.596899 13041 log.cc:1196] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/2355a2f350f24fe4b607e140dbe8b307
I20260620 18:34:24.597128 13041 ts_tablet_manager.cc:1939] T 2355a2f350f24fe4b607e140dbe8b307 P 40555eab05f94e348d23b18db8312f92: Deleting consensus metadata
I20260620 18:34:24.597499 12388 catalog_manager.cc:4841] TS 40555eab05f94e348d23b18db8312f92 (127.10.109.67:42629): tablet 2355a2f350f24fe4b607e140dbe8b307 (table test-table [id=0caee372ced446bd839d92449fd8e75b]) successfully deleted
W20260620 18:34:24.598186 12391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:37095 (error 108)
I20260620 18:34:24.598423 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12616
I20260620 18:34:24.599126 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:24.598547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37095 (local address 127.0.0.1:53918)
0620 18:34:24.599058 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:24.599096 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:37095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":56,"threads_started":1}
W20260620 18:34:24.599231 12391 catalog_manager.cc:4584] TS 4b49a46884144c94973980f1b8a39009 (127.10.109.65:37095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2355a2f350f24fe4b607e140dbe8b307: Network error: Client connection negotiation failed: client connection to 127.10.109.65:37095: connect: Connection refused (error 111)
I20260620 18:34:24.605005 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12756
I20260620 18:34:24.610337 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 12372
2026-06-20T18:34:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5575 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:24 dist-test-slave-3jql krb5kdc[13048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:24 dist-test-slave-3jql krb5kdc[13048](info): set up 2 sockets
Jun 20 18:34:24 dist-test-slave-3jql krb5kdc[13048](info): commencing operation
krb5kdc: starting...
W20260620 18:34:25.663157 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:25 dist-test-slave-3jql krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:25.779992 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36569
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:25.860389 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:25.860595 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:25.860618 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:25.861754 13061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:25.861794 13061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:25.861809 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:25.861821 13061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:25.861835 13061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:25.863375 13061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:25.863626 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:25.863909 13061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:25.869305 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.869341 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:25.869410 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:25.869443 13061 server_base.cc:1034] running on GCE node
I20260620 18:34:25.869863 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:25.870177 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:25.871371 13061 hybrid_clock.cc:648] HybridClock initialized: now 1781980465871345 us; error 43 us; skew 500 ppm
Jun 20 18:34:25 dist-test-slave-3jql krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:25.874115 13061 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:25.874545 13061 webserver.cc:460] Webserver started at http://127.10.109.126:39799/ using document root <none> and password file <none>
I20260620 18:34:25.874838 13061 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:25.874907 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:25.875073 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:25.876104 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "b9567862371142bda06d7a356bc7f91d"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-3jql"
server_key: "ff75342c7c9f755ac6c0d763ae16dc98"
server_key_iv: "0562e3ccf440adc7d78fddad69f9dadc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.876471 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "b9567862371142bda06d7a356bc7f91d"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-3jql"
server_key: "ff75342c7c9f755ac6c0d763ae16dc98"
server_key_iv: "0562e3ccf440adc7d78fddad69f9dadc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.877782 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:25.878552 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:25.878769 13061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:25.878857 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "b9567862371142bda06d7a356bc7f91d"
format_stamp: "Formatted at 2026-06-20 18:34:25 on dist-test-slave-3jql"
server_key: "ff75342c7c9f755ac6c0d763ae16dc98"
server_key_iv: "0562e3ccf440adc7d78fddad69f9dadc"
server_key_version: "encryptionkey@0"
I20260620 18:34:25.878969 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:25.917299 13061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:25.919672 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:25.919843 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:25.923985 13061 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36569
I20260620 18:34:25.924118 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36569 every 8 connection(s)
I20260620 18:34:25.924402 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:25.924870 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13061
I20260620 18:34:25.924955 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:25.925143 10677 external_mini_cluster.cc:1428] Setting key d55f1e0656b55f70eceafd49843cf6b2
I20260620 18:34:25.925423 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:25.928879 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d: Bootstrap starting.
I20260620 18:34:25.929481 13129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:25.929800 13129 log.cc:824] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:25.930621 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d: No bootstrap required, opened a new log
Jun 20 18:34:25 dist-test-slave-3jql krb5kdc[13048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:25.932600 13129 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } }
I20260620 18:34:25.932778 13129 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:25.932830 13129 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9567862371142bda06d7a356bc7f91d, State: Initialized, Role: FOLLOWER
I20260620 18:34:25.933002 13129 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } }
I20260620 18:34:25.933112 13129 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:25.933147 13129 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:25.933205 13129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:25.933920 13129 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } }
I20260620 18:34:25.934032 13129 leader_election.cc:302] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9567862371142bda06d7a356bc7f91d; no voters:
I20260620 18:34:25.934204 13129 leader_election.cc:288] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:25.934262 13134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:25.934448 13134 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [term 1 LEADER]: Becoming Leader. State: Replica: b9567862371142bda06d7a356bc7f91d, State: Running, Role: LEADER
I20260620 18:34:25.934453 13129 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:25.934585 13134 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } }
I20260620 18:34:25.935043 13135 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9567862371142bda06d7a356bc7f91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } } }
I20260620 18:34:25.935123 13135 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:25.935277 13136 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9567862371142bda06d7a356bc7f91d. Latest consensus state: current_term: 1 leader_uuid: "b9567862371142bda06d7a356bc7f91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9567862371142bda06d7a356bc7f91d" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36569 } } }
I20260620 18:34:25.935371 13136 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:25.935770 13138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:25.936252 13138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:25.936470 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:25.925686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.10.109.126:36569)
0620 18:34:25.925875 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:25.925884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:25.925915 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:34:25.926475 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:25.926484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:25.926547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:25.926711 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:25.927361 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.928726 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.929559 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:25.930148 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:25.930233 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:25.932523 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:25.932528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:25.932528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:25.932540 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:25.934662 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:25.935081 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:25.935084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:25.935085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:25.935185 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:25.935424 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:25.935425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:25.935425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:25.935770 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:25.935921 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:25.936048 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:34:25.936196 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260620 18:34:25.938328 13138 catalog_manager.cc:1347] Generated new cluster ID: a5b7603cd9de4e079bfeddec3751e3dd
I20260620 18:34:25.938402 13138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:25.978937 13138 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:25.979528 13138 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:25.985330 13138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9567862371142bda06d7a356bc7f91d: Generated new TSK 0
I20260620 18:34:25.985518 13138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:26.025394 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:26.106662 13155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:26.106873 13155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:26.106895 13155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:26.107985 13155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:26.108017 13155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:26.108042 13155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:26.109338 13155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:26.109586 13155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:26.109750 13155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:26.114756 13160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.114770 13163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.114795 13161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:26.115221 13155 server_base.cc:1034] running on GCE node
I20260620 18:34:26.115403 13155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:26.115585 13155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:26.116771 13155 hybrid_clock.cc:648] HybridClock initialized: now 1781980466116762 us; error 40 us; skew 500 ppm
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:26.119331 13155 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:26.119927 13155 webserver.cc:460] Webserver started at http://127.10.109.65:45567/ using document root <none> and password file <none>
I20260620 18:34:26.120216 13155 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:26.120277 13155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:26.120420 13155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:26.121585 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "cf5b661cd89e4020b415c02908f1726d"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "4ee1a70387141769eea1b4035ee92ca2"
server_key_iv: "95e0333d4ee49dfd17b84804d02591f2"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.121935 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "cf5b661cd89e4020b415c02908f1726d"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "4ee1a70387141769eea1b4035ee92ca2"
server_key_iv: "95e0333d4ee49dfd17b84804d02591f2"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.123356 13155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:26.124105 13170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.124310 13155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:26.124405 13155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "cf5b661cd89e4020b415c02908f1726d"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "4ee1a70387141769eea1b4035ee92ca2"
server_key_iv: "95e0333d4ee49dfd17b84804d02591f2"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.124497 13155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:26.139249 13155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:26.141471 13155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:26.141625 13155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:26.141844 13155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:26.142144 13155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:26.142195 13155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.142236 13155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:26.142264 13155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.148479 13155 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:35933
I20260620 18:34:26.148517 13283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:35933 every 8 connection(s)
I20260620 18:34:26.148849 13155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:26.150066 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13155
I20260620 18:34:26.150139 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:26.150312 10677 external_mini_cluster.cc:1428] Setting key 64cb8d29ad3e3d43c48b9e2974c30688
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
I20260620 18:34:26.158901 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.149552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:53637 (local address 127.10.109.126:36569)
0620 18:34:26.149626 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.149629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.149832 (+ 203us) server_negotiation.cc:407] Connection header received
0620 18:34:26.150460 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.150462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.150473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.150518 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.151565 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.152075 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.153054 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.153276 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.153363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.155620 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.155621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.155621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:26.155630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.157395 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.157785 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.157786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.157787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.157849 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.158148 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.158149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.158149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.158289 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.158371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.158706 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.158825 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:34:26.159041 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.149524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36569 (local address 127.10.109.65:53637)
0620 18:34:26.149710 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.149743 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.150354 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.150620 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.150625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.151280 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:26.151460 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.151465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.152171 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.152172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.152961 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.152964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.153372 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.153374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.153495 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.154032 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:26.154040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:26.155523 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:26.157493 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.157499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.157501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.157704 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.157952 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.157953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.157953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.158071 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.158433 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:26.158440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:26.158564 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.158658 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.158836 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260620 18:34:26.159535 13284 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36569
I20260620 18:34:26.159644 13284 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:26.159814 13284 heartbeater.cc:511] Master 127.10.109.126:36569 requested a full tablet report, sending...
I20260620 18:34:26.160394 13093 ts_manager.cc:194] Registered new tserver with Master: cf5b661cd89e4020b415c02908f1726d (127.10.109.65:35933)
I20260620 18:34:26.161019 13093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:53637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:26.178270 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:26.256925 13289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:26.257128 13289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:26.257150 13289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:26.258391 13289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:26.258450 13289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:26.258481 13289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:26.260205 13289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:26.260403 13289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:26.260582 13289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:26.266063 13295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.266072 13294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:26.266103 13289 server_base.cc:1034] running on GCE node
W20260620 18:34:26.266027 13297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:26.266384 13289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:26.266644 13289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:26.267813 13289 hybrid_clock.cc:648] HybridClock initialized: now 1781980466267793 us; error 32 us; skew 500 ppm
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:26.270289 13289 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:26.270653 13289 webserver.cc:460] Webserver started at http://127.10.109.66:39345/ using document root <none> and password file <none>
I20260620 18:34:26.270900 13289 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:26.270947 13289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:26.271034 13289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:26.271970 13289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "a81c86590e4c48188e362a2f054ef4ae"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "96a3c92ae86daf7c2bb1e21e973d95f0"
server_key_iv: "aad0bd0920eb90a92ae34fb07ab7522e"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.272308 13289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "a81c86590e4c48188e362a2f054ef4ae"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "96a3c92ae86daf7c2bb1e21e973d95f0"
server_key_iv: "aad0bd0920eb90a92ae34fb07ab7522e"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.273532 13289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:26.274288 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.274461 13289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:26.274542 13289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "a81c86590e4c48188e362a2f054ef4ae"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "96a3c92ae86daf7c2bb1e21e973d95f0"
server_key_iv: "aad0bd0920eb90a92ae34fb07ab7522e"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.274596 13289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:26.287206 13289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:26.288935 13289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:26.289036 13289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:26.289201 13289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:26.289434 13289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:26.289469 13289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.289489 13289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:26.289500 13289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.295229 13289 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:41063
I20260620 18:34:26.295297 13417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:41063 every 8 connection(s)
I20260620 18:34:26.295590 13289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:26.302623 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13289
I20260620 18:34:26.302737 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:26.302958 10677 external_mini_cluster.cc:1428] Setting key bc89e300c2478556019bc834bd17bfda
I20260620 18:34:26.304955 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.296260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:52027 (local address 127.10.109.126:36569)
0620 18:34:26.296346 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.296348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.296524 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:34:26.296927 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.296928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.296939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.296992 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.297864 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.298373 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.299555 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.299737 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.299795 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.301732 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.301733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.301733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:26.301739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.303397 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.303835 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.303836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.303836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.303885 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.304149 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.304149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.304150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.304244 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.304294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.304830 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.304895 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:34:26.305044 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.296213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36569 (local address 127.10.109.66:52027)
0620 18:34:26.296387 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.296419 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.296845 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.297038 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.297041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.297602 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:26.297776 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.297780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.298468 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.298469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.299455 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.299457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.299816 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.299817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.299913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.300341 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:26.300349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:26.301651 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:26.303461 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.303467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.303469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.303755 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.303950 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.303951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.303951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.304082 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.304307 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:26.304312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:26.304477 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.304610 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.304777 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260620 18:34:26.305833 13418 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36569
I20260620 18:34:26.305909 13418 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:26.306097 13418 heartbeater.cc:511] Master 127.10.109.126:36569 requested a full tablet report, sending...
I20260620 18:34:26.306456 13093 ts_manager.cc:194] Registered new tserver with Master: a81c86590e4c48188e362a2f054ef4ae (127.10.109.66:41063)
I20260620 18:34:26.306922 13093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:52027
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:26.330233 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:26.410857 13423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:26.411062 13423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:26.411085 13423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:26.412245 13423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:26.412286 13423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:26.412313 13423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:26.413712 13423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:26.413913 13423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:26.414124 13423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:26.419062 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.419142 13428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:26.419189 13431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:26.419553 13423 server_base.cc:1034] running on GCE node
I20260620 18:34:26.419741 13423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:26.419925 13423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:26.421061 13423 hybrid_clock.cc:648] HybridClock initialized: now 1781980466421051 us; error 32 us; skew 500 ppm
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:26.424067 13423 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:26.424552 13423 webserver.cc:460] Webserver started at http://127.10.109.67:39489/ using document root <none> and password file <none>
I20260620 18:34:26.424775 13423 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:26.424818 13423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:26.424948 13423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:26.426000 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "b04e10ce816a46bdbd63e5c7063f229a"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "e2e13362beca61b0de75c46ac044d8c9"
server_key_iv: "f7b1c8d4d6054daca8fd799801f00eb6"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.426347 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "b04e10ce816a46bdbd63e5c7063f229a"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "e2e13362beca61b0de75c46ac044d8c9"
server_key_iv: "f7b1c8d4d6054daca8fd799801f00eb6"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.427817 13423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:26.428566 13438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.428884 13423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:26.428992 13423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "b04e10ce816a46bdbd63e5c7063f229a"
format_stamp: "Formatted at 2026-06-20 18:34:26 on dist-test-slave-3jql"
server_key: "e2e13362beca61b0de75c46ac044d8c9"
server_key_iv: "f7b1c8d4d6054daca8fd799801f00eb6"
server_key_version: "encryptionkey@0"
I20260620 18:34:26.429076 13423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:26.460275 13423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:26.462090 13423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:26.462246 13423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:26.462478 13423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:26.462833 13423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:26.462884 13423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.462926 13423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:26.462954 13423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:26.468665 13423 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40351
I20260620 18:34:26.468716 13551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40351 every 8 connection(s)
I20260620 18:34:26.469043 13423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:26.475034 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13423
I20260620 18:34:26.475124 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:26.475302 10677 external_mini_cluster.cc:1428] Setting key c8cb194894e04b9af45fee40ea6ef2e3
I20260620 18:34:26.478044 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.469709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:46495 (local address 127.10.109.126:36569)
0620 18:34:26.469773 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.469775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.469987 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:34:26.470436 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.470437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.470446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.470504 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.471467 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.471945 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.472895 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.473093 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.473151 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.475377 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:26.475379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:26.475379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:26.475386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:26.476858 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.477303 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.477304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.477304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.477352 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:26.477558 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:26.477559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:26.477559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:26.477673 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:26.477740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:26.477931 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:34:26.477985 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:34:26.478232 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:26.469701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36569 (local address 127.10.109.67:46495)
0620 18:34:26.469862 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:26.469895 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:26.470352 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:26.470577 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:26.470580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:26.471207 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:26.471385 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.471390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.472040 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.472040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.472820 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:26.472822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:26.473191 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:26.473192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:26.473296 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:26.473838 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:26.473846 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:26.475253 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:26.476931 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.476936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.476938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.477230 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.477408 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:26.477408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:26.477408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:26.477504 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:26.477734 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:26.477738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:26.477838 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 18:34:26.477908 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:34:26.478033 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260620 18:34:26.478770 13552 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36569
I20260620 18:34:26.478874 13552 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:26.479075 13552 heartbeater.cc:511] Master 127.10.109.126:36569 requested a full tablet report, sending...
I20260620 18:34:26.479426 13093 ts_manager.cc:194] Registered new tserver with Master: b04e10ce816a46bdbd63e5c7063f229a (127.10.109.67:40351)
I20260620 18:34:26.479794 13093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:46495
I20260620 18:34:26.486853 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:34:26.496953 13132 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:26.496176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.10.109.126:36569)
0620 18:34:26.496261 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:26.496263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:26.496314 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:34:26.496354 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:26.496355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:26.496363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:26.496415 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:26.496891 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55070: BlockingRecv error: recv got EOF from 127.0.0.1:55070 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:26.497920 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13155
I20260620 18:34:26.502380 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13289
I20260620 18:34:26.506703 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13423
I20260620 18:34:26.510727 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13061
2026-06-20T18:34:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1899 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13564](info): set up 2 sockets
Jun 20 18:34:26 dist-test-slave-3jql krb5kdc[13564](info): commencing operation
krb5kdc: starting...
W20260620 18:34:27.563083 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:27.673688 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36077
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:27.752170 13577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:27.752398 13577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:27.752434 13577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:27.753554 13577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:27.753628 13577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:27.753654 13577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:27.753681 13577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:27.753713 13577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:27.755182 13577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:27.755440 13577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:27.755656 13577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:27.760282 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:27.760315 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:27.760282 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:27.760445 13577 server_base.cc:1034] running on GCE node
I20260620 18:34:27.760807 13577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:27.761085 13577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:27.762243 13577 hybrid_clock.cc:648] HybridClock initialized: now 1781980467762228 us; error 31 us; skew 500 ppm
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:27.764616 13577 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:27.765064 13577 webserver.cc:460] Webserver started at http://127.10.109.126:42775/ using document root <none> and password file <none>
I20260620 18:34:27.765287 13577 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:27.765353 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:27.765503 13577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:27.766477 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "e75de78393e44077aedd533b37d6e3cd"
format_stamp: "Formatted at 2026-06-20 18:34:27 on dist-test-slave-3jql"
server_key: "7cf43542f74345fa7294aaabeb360e75"
server_key_iv: "031736dd89cfd3ad4a1530ba760cad0a"
server_key_version: "encryptionkey@0"
I20260620 18:34:27.766904 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "e75de78393e44077aedd533b37d6e3cd"
format_stamp: "Formatted at 2026-06-20 18:34:27 on dist-test-slave-3jql"
server_key: "7cf43542f74345fa7294aaabeb360e75"
server_key_iv: "031736dd89cfd3ad4a1530ba760cad0a"
server_key_version: "encryptionkey@0"
I20260620 18:34:27.768210 13577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:27.768961 13592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:27.769145 13577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:27.769243 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "e75de78393e44077aedd533b37d6e3cd"
format_stamp: "Formatted at 2026-06-20 18:34:27 on dist-test-slave-3jql"
server_key: "7cf43542f74345fa7294aaabeb360e75"
server_key_iv: "031736dd89cfd3ad4a1530ba760cad0a"
server_key_version: "encryptionkey@0"
I20260620 18:34:27.769330 13577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:27.779044 13577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:27.780856 13577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:27.781013 13577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:27.785123 13577 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36077
I20260620 18:34:27.785168 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36077 every 8 connection(s)
I20260620 18:34:27.785651 13577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:27.786267 13645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:27.787465 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13577
I20260620 18:34:27.787545 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:27.787739 10677 external_mini_cluster.cc:1428] Setting key 56de1f68dd696fd058be8081c11c245f
I20260620 18:34:27.788897 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd: Bootstrap starting.
I20260620 18:34:27.789667 13645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:27.789980 13645 log.cc:824] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:27.790661 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd: No bootstrap required, opened a new log
I20260620 18:34:27.793087 13645 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } }
I20260620 18:34:27.793260 13645 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:27.793303 13645 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75de78393e44077aedd533b37d6e3cd, State: Initialized, Role: FOLLOWER
I20260620 18:34:27.793457 13645 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } }
I20260620 18:34:27.793548 13645 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:27.793584 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:27.793634 13645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:27.794253 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } }
I20260620 18:34:27.794339 13645 leader_election.cc:302] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75de78393e44077aedd533b37d6e3cd; no voters:
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:27.794544 13645 leader_election.cc:288] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:27.794567 13650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:27.794683 13650 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [term 1 LEADER]: Becoming Leader. State: Replica: e75de78393e44077aedd533b37d6e3cd, State: Running, Role: LEADER
I20260620 18:34:27.794806 13645 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:27.794790 13650 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } }
I20260620 18:34:27.795111 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75de78393e44077aedd533b37d6e3cd. Latest consensus state: current_term: 1 leader_uuid: "e75de78393e44077aedd533b37d6e3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } } }
I20260620 18:34:27.795184 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:27.795385 13655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:27.795573 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75de78393e44077aedd533b37d6e3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75de78393e44077aedd533b37d6e3cd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36077 } } }
I20260620 18:34:27.795671 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:27.795897 13655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:27.797632 13655 catalog_manager.cc:1347] Generated new cluster ID: 271e29a4228f4e359ec2a377f0eb194c
I20260620 18:34:27.797685 13655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:27.799136 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.788201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.10.109.126:36077)
0620 18:34:27.788368 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:27.788375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:27.788398 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:34:27.788782 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:27.788788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:27.788841 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:27.788951 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:27.789754 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.790899 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.791920 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.792292 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.792362 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:27.795114 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:27.795118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:27.795118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:27.795128 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:27.797427 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:27.797926 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:27.797929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:27.797930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:27.798025 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:27.798248 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:27.798249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:27.798249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:27.798479 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:27.798617 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:27.798762 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:34:27.798899 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260620 18:34:27.807456 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:27.812254 13655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:27.812933 13655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:27.820253 13655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75de78393e44077aedd533b37d6e3cd: Generated new TSK 0
I20260620 18:34:27.820428 13655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:27.823619 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:27.819488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.10.109.126:36077)
0620 18:34:27.819529 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:27.819531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:27.819657 (+ 126us) server_negotiation.cc:407] Connection header received
0620 18:34:27.819714 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:27.819715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:27.819726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:27.819778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:27.820251 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.821332 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.822100 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:27.822428 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:27.822500 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:27.823163 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:27.823165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:27.823165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:27.823373 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:27.823392 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:27.823398 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:27.823463 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:27.823510 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:34:27.823536 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:27.825793 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13577
2026-06-20T18:34:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1312 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13678](info): set up 2 sockets
Jun 20 18:34:27 dist-test-slave-3jql krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260620 18:34:28.887348 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:28 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:29.010850 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45003
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:29.095078 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:29.095311 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:29.095350 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:29.096446 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:29.096527 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:29.096553 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:29.096583 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:29.096606 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:29.097994 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:29.098255 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:29.098472 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.103196 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.103251 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.103251 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.103542 13691 server_base.cc:1034] running on GCE node
I20260620 18:34:29.103785 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.104056 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.105240 13691 hybrid_clock.cc:648] HybridClock initialized: now 1781980469105233 us; error 52 us; skew 500 ppm
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.107656 13691 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:29.108114 13691 webserver.cc:460] Webserver started at http://127.10.109.126:38237/ using document root <none> and password file <none>
I20260620 18:34:29.108330 13691 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.108397 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.108543 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.109526 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "080f190b41d04b27831b1581c4cc6c98"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "c81e6f34fa254a9a93b302a1a3c63f6e"
server_key_iv: "29eea496c96bf883d4a97237832129fa"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.109875 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "080f190b41d04b27831b1581c4cc6c98"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "c81e6f34fa254a9a93b302a1a3c63f6e"
server_key_iv: "29eea496c96bf883d4a97237832129fa"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.111258 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:29.112004 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.112175 13691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.112270 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "080f190b41d04b27831b1581c4cc6c98"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "c81e6f34fa254a9a93b302a1a3c63f6e"
server_key_iv: "29eea496c96bf883d4a97237832129fa"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.112347 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.148160 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.150251 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.150418 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.154436 13691 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:45003
I20260620 18:34:29.154491 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:45003 every 8 connection(s)
I20260620 18:34:29.154865 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:29.155540 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13691
I20260620 18:34:29.155627 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:29.155822 10677 external_mini_cluster.cc:1428] Setting key e234451ed00f60b0b999288b89ec1544
I20260620 18:34:29.155941 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:29.159301 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: Bootstrap starting.
I20260620 18:34:29.159916 13759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:29.160209 13759 log.cc:824] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:29.160962 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: No bootstrap required, opened a new log
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:29.162931 13759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } }
I20260620 18:34:29.163087 13759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:29.163127 13759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080f190b41d04b27831b1581c4cc6c98, State: Initialized, Role: FOLLOWER
I20260620 18:34:29.163264 13759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } }
I20260620 18:34:29.163379 13759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:29.163434 13759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:29.163484 13759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:29.164093 13759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } }
I20260620 18:34:29.164211 13759 leader_election.cc:302] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080f190b41d04b27831b1581c4cc6c98; no voters:
I20260620 18:34:29.164366 13759 leader_election.cc:288] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:29.164578 13764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:29.164649 13759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:29.164733 13764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [term 1 LEADER]: Becoming Leader. State: Replica: 080f190b41d04b27831b1581c4cc6c98, State: Running, Role: LEADER
I20260620 18:34:29.164836 13764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } }
I20260620 18:34:29.165921 13766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "080f190b41d04b27831b1581c4cc6c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } } }
I20260620 18:34:29.166002 13766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:29.166299 13765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080f190b41d04b27831b1581c4cc6c98. Latest consensus state: current_term: 1 leader_uuid: "080f190b41d04b27831b1581c4cc6c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080f190b41d04b27831b1581c4cc6c98" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45003 } } }
I20260620 18:34:29.166363 13765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98 [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:29.167052 13780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:29.167141 13780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:29.167212 13774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:29.167392 13774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:29.168457 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.156312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.10.109.126:45003)
0620 18:34:29.156496 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.156506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.156542 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:34:29.157179 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.157188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.157248 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.157399 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.158076 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.159444 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.160325 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.160835 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.160959 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.163923 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.163927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.163927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.163939 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.166592 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.167132 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.167135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.167136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.167254 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.167494 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.167495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.167496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.167729 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.167827 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.168040 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.168268 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260620 18:34:29.168963 13774 catalog_manager.cc:1347] Generated new cluster ID: 4fccabdf6ea04b859f30fac2d2f98b95
I20260620 18:34:29.169021 13774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:29.179648 13774 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:29.180159 13774 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:29.183239 13774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080f190b41d04b27831b1581c4cc6c98: Generated new TSK 0
I20260620 18:34:29.183390 13774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:29.214383 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:29.292987 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:29.293215 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:29.293251 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:29.294358 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:29.294425 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:29.294468 13785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:29.295938 13785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:29.296195 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:29.296422 13785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.301724 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.301765 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.301842 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.302075 13785 server_base.cc:1034] running on GCE node
I20260620 18:34:29.302287 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.302529 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.303701 13785 hybrid_clock.cc:648] HybridClock initialized: now 1781980469303680 us; error 39 us; skew 500 ppm
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.306389 13785 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:29.306953 13785 webserver.cc:460] Webserver started at http://127.10.109.65:41853/ using document root <none> and password file <none>
I20260620 18:34:29.307214 13785 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.307287 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.307466 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.308594 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "b9b886fe00494ca9b5ba7d12019e5c3e"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "4040abaaefde7828d840e758b62c4b8e"
server_key_iv: "23d261c77a75bacbc31cf13dea6a2c2c"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.308988 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "b9b886fe00494ca9b5ba7d12019e5c3e"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "4040abaaefde7828d840e758b62c4b8e"
server_key_iv: "23d261c77a75bacbc31cf13dea6a2c2c"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.310376 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:29.311292 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.311512 13785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:29.311609 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "b9b886fe00494ca9b5ba7d12019e5c3e"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "4040abaaefde7828d840e758b62c4b8e"
server_key_iv: "23d261c77a75bacbc31cf13dea6a2c2c"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.311697 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.331883 13785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.334358 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.334537 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.334827 13785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:29.335158 13785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:29.335218 13785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.335263 13785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:29.335295 13785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.341372 13785 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:44321
I20260620 18:34:29.341444 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:44321 every 8 connection(s)
I20260620 18:34:29.341787 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:29.349309 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13785
I20260620 18:34:29.349423 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:29.349638 10677 external_mini_cluster.cc:1428] Setting key 6a6a8180c5f45202f26acd729c0661a4
I20260620 18:34:29.351467 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.342600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:38735 (local address 127.10.109.126:45003)
0620 18:34:29.342666 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.342668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.342926 (+ 258us) server_negotiation.cc:407] Connection header received
0620 18:34:29.343439 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.343440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.343449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.343513 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.344434 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.344930 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.345951 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.346146 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.346200 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.348389 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.348390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.348390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.348397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.349859 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.350327 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.350328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.350329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.350402 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.350704 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.350705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.350705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.350903 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.350995 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.351269 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.351358 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:34:29.351800 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.342580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45003 (local address 127.10.109.65:38735)
0620 18:34:29.342773 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:29.342817 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:29.343353 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:29.343568 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:29.343571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:29.344155 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:29.344354 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.344358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.345014 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.345014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.345867 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.345870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.346223 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.346224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.346327 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.346929 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:29.346938 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:29.348286 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:29.349956 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.349963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.349965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.350244 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.350494 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.350495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.350496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.350616 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.350982 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:29.350989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:29.351127 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:34:29.351247 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:34:29.351443 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260620 18:34:29.352295 13914 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45003
I20260620 18:34:29.352392 13914 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:29.352591 13914 heartbeater.cc:511] Master 127.10.109.126:45003 requested a full tablet report, sending...
I20260620 18:34:29.353132 13723 ts_manager.cc:194] Registered new tserver with Master: b9b886fe00494ca9b5ba7d12019e5c3e (127.10.109.65:44321)
I20260620 18:34:29.353780 13723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:38735
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:29.378840 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:29.460502 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:29.460722 13919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:29.460748 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:29.461933 13919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:29.461971 13919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:29.461999 13919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:29.463670 13919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:29.463868 13919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:29.464121 13919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.469739 13924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.469751 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.469933 13927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.470276 13919 server_base.cc:1034] running on GCE node
I20260620 18:34:29.470502 13919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.470701 13919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.471913 13919 hybrid_clock.cc:648] HybridClock initialized: now 1781980469471891 us; error 35 us; skew 500 ppm
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.474475 13919 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:29.474957 13919 webserver.cc:460] Webserver started at http://127.10.109.66:36405/ using document root <none> and password file <none>
I20260620 18:34:29.475189 13919 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.475234 13919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.475320 13919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.476263 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "43a4fc3925f940bf983dce10e42bb278"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "89072ffc4dbfaa5ecb304c109eb9c6e1"
server_key_iv: "a48df569bf9c65e4e6d8f0efb219f855"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.476558 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "43a4fc3925f940bf983dce10e42bb278"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "89072ffc4dbfaa5ecb304c109eb9c6e1"
server_key_iv: "a48df569bf9c65e4e6d8f0efb219f855"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.477837 13919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:34:29.478538 13934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.478756 13919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:29.478830 13919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "43a4fc3925f940bf983dce10e42bb278"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "89072ffc4dbfaa5ecb304c109eb9c6e1"
server_key_iv: "a48df569bf9c65e4e6d8f0efb219f855"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.478881 13919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.493989 13919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.496268 13919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.496373 13919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.496546 13919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:29.496785 13919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:29.496848 13919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.496896 13919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:29.496912 13919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.502997 13919 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:40775
I20260620 18:34:29.503074 14047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:40775 every 8 connection(s)
I20260620 18:34:29.503366 13919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:29.503643 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 13919
I20260620 18:34:29.503759 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:29.503975 10677 external_mini_cluster.cc:1428] Setting key a32d05d667958074e11a663ab493eccb
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:34:29.514426 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.504116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:59995 (local address 127.10.109.126:45003)
0620 18:34:29.504163 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.504165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.504394 (+ 229us) server_negotiation.cc:407] Connection header received
0620 18:34:29.504903 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.504905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.504916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.504980 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.505966 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.506549 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.508126 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.508339 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.508400 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.510909 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.510911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.510911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.510919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.512828 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.513348 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.513350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.513350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.513414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.513676 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.513677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.513677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.513801 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.513930 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.514207 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.514287 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:29.514585 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.504115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45003 (local address 127.10.109.66:59995)
0620 18:34:29.504285 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:29.504324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:29.504804 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:29.505050 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:29.505054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:29.505609 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:29.505833 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.505839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.506653 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.506655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.508015 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.508019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.508428 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.508429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.508590 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.509133 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:29.509144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:29.510805 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:29.513001 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.513009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.513011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.513254 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.513487 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.513488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.513489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.513604 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.513907 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:29.513913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:29.514068 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 18:34:29.514184 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:34:29.514344 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
I20260620 18:34:29.515146 14048 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45003
I20260620 18:34:29.515254 14048 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:29.515473 14048 heartbeater.cc:511] Master 127.10.109.126:45003 requested a full tablet report, sending...
I20260620 18:34:29.515930 13723 ts_manager.cc:194] Registered new tserver with Master: 43a4fc3925f940bf983dce10e42bb278 (127.10.109.66:40775)
I20260620 18:34:29.516503 13723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:59995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:29.531780 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:29.610059 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:29.610242 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:29.610265 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:29.611308 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:29.611342 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:29.611367 14053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:29.612665 14053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:29.612835 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:29.612986 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:29.617843 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:29.617846 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.617937 14053 server_base.cc:1034] running on GCE node
W20260620 18:34:29.617851 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:29.618218 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:29.618449 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:29.619612 14053 hybrid_clock.cc:648] HybridClock initialized: now 1781980469619598 us; error 26 us; skew 500 ppm
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:29.622108 14053 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:29.622514 14053 webserver.cc:460] Webserver started at http://127.10.109.67:43051/ using document root <none> and password file <none>
I20260620 18:34:29.622777 14053 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:29.622854 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:29.623008 14053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:29.623973 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "a7355e99af4547d0bf7782a5635da18b"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "78120ecb5a8f926334b1d13cfee02501"
server_key_iv: "74fbd8960581bdbbdbcb5dddd2f657d4"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.624331 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "a7355e99af4547d0bf7782a5635da18b"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "78120ecb5a8f926334b1d13cfee02501"
server_key_iv: "74fbd8960581bdbbdbcb5dddd2f657d4"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.625635 14053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:29.626384 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.626557 14053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:29.626654 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "a7355e99af4547d0bf7782a5635da18b"
format_stamp: "Formatted at 2026-06-20 18:34:29 on dist-test-slave-3jql"
server_key: "78120ecb5a8f926334b1d13cfee02501"
server_key_iv: "74fbd8960581bdbbdbcb5dddd2f657d4"
server_key_version: "encryptionkey@0"
I20260620 18:34:29.626739 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:29.637468 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:29.639662 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:29.639815 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:29.640020 14053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:29.640305 14053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:29.640358 14053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.640398 14053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:29.640426 14053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:29.646783 14053 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40095
I20260620 18:34:29.646862 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40095 every 8 connection(s)
I20260620 18:34:29.647203 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:29.656364 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14053
I20260620 18:34:29.656525 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:29.656746 10677 external_mini_cluster.cc:1428] Setting key 523824e170a5b8491e9bfb16d4ca0f2b
I20260620 18:34:29.658215 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.647947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:36473 (local address 127.10.109.126:45003)
0620 18:34:29.648050 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.648053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.648244 (+ 191us) server_negotiation.cc:407] Connection header received
0620 18:34:29.648750 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.648751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.648761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.648821 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.649676 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.650182 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.651853 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.652084 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.652157 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.654649 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.654650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.654650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.654658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.656295 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.656959 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.656961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.656962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.657039 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.657431 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.657433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.657434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.657609 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.657663 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.658072 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.658149 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:34:29.658526 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.647901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45003 (local address 127.10.109.67:36473)
0620 18:34:29.648118 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:29.648157 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:29.648647 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:29.648870 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:29.648873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:29.649409 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:29.649600 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.649604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.650362 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.650365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.651664 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:29.651669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.652234 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.652236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:29.652423 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.652921 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:29.652929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:29.654528 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:29.656463 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.656472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.656474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.656800 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.657150 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:29.657151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:29.657152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:29.657331 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:29.657718 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:29.657724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:29.657921 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 18:34:29.658018 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:34:29.658223 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":147,"threads_started":1}
I20260620 18:34:29.659215 14182 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45003
I20260620 18:34:29.659343 14182 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:29.659543 14182 heartbeater.cc:511] Master 127.10.109.126:45003 requested a full tablet report, sending...
I20260620 18:34:29.659996 13723 ts_manager.cc:194] Registered new tserver with Master: a7355e99af4547d0bf7782a5635da18b (127.10.109.67:40095)
I20260620 18:34:29.660400 13723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:36473
I20260620 18:34:29.668637 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980469, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:29.695153 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:29.683968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.10.109.126:45003)
0620 18:34:29.684026 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:29.684028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:29.684142 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:34:29.684187 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:29.684188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:29.684197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:29.684255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:29.684753 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.685225 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.685934 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:29.686104 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:29.686151 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:29.692751 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:29.692752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:29.692753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:29.692759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:29.694098 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.694420 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.694421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.694421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.694468 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:29.694634 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:29.694635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:29.694635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:29.694834 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:29.694914 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:29.695026 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:34:29.695091 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 18:34:29.695387 13723 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36990
I20260620 18:34:29.695796 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13785
I20260620 18:34:29.700227 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13919
I20260620 18:34:29.705168 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14053
I20260620 18:34:29.709928 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 13691
2026-06-20T18:34:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1886 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[14194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[14194](info): set up 2 sockets
Jun 20 18:34:29 dist-test-slave-3jql krb5kdc[14194](info): commencing operation
krb5kdc: starting...
W20260620 18:34:30.836556 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 18:34:31.135200 10677 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.299s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 18:34:31.408560 10677 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.273s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 18:34:31.710868 10677 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.302s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/test-user.keytab.
W20260620 18:34:32.460078 10677 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.749s user 0.001s sys 0.001s
Jun 20 18:34:32 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
W20260620 18:34:33.699405 10677 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.109.126: real 1.220s user 0.001s sys 0.002s
I20260620 18:34:33.699833 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42263
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:33.793880 14207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:33.794116 14207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:33.794142 14207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:33.795557 14207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:33.795614 14207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:33.795643 14207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:33.795670 14207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:33.795706 14207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:33.797586 14207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:33.797824 14207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:33.798041 14207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:33.804788 14215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.804832 14213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:33.804809 14212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:33.805187 14207 server_base.cc:1034] running on GCE node
I20260620 18:34:33.805457 14207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:33.805701 14207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:33.806862 14207 hybrid_clock.cc:648] HybridClock initialized: now 1781980473806834 us; error 29 us; skew 500 ppm
Jun 20 18:34:33 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980473, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:33.809721 14207 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:33.810153 14207 webserver.cc:460] Webserver started at http://127.10.109.126:35227/ using document root <none> and password file <none>
I20260620 18:34:33.810396 14207 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:33.810467 14207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:33.810613 14207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:33.811724 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "0201b8139def41639f959d0e96260c82"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-3jql"
server_key: "c8a59c3d1d386984d9ba3c23d215cd93"
server_key_iv: "03d32b47977627e8af7dbc5dac00580a"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.812100 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "0201b8139def41639f959d0e96260c82"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-3jql"
server_key: "c8a59c3d1d386984d9ba3c23d215cd93"
server_key_iv: "03d32b47977627e8af7dbc5dac00580a"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.813524 14207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:33.814301 14222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:33.814563 14207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:33.814669 14207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "0201b8139def41639f959d0e96260c82"
format_stamp: "Formatted at 2026-06-20 18:34:33 on dist-test-slave-3jql"
server_key: "c8a59c3d1d386984d9ba3c23d215cd93"
server_key_iv: "03d32b47977627e8af7dbc5dac00580a"
server_key_version: "encryptionkey@0"
I20260620 18:34:33.814774 14207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:33.827955 14207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:33.829867 14207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:33.830025 14207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:33.833984 14207 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:42263
I20260620 18:34:33.834017 14274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:42263 every 8 connection(s)
I20260620 18:34:33.834506 14207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:33.835263 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14207
I20260620 18:34:33.835290 14275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:33.835397 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:33.835574 10677 external_mini_cluster.cc:1428] Setting key e28fb617371243aef3901609f83fe7b9
I20260620 18:34:33.838433 14275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82: Bootstrap starting.
I20260620 18:34:33.839141 14275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:33.839455 14275 log.cc:824] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:33.840164 14275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82: No bootstrap required, opened a new log
I20260620 18:34:33.842607 14275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } }
I20260620 18:34:33.842732 14275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:33.842854 14275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0201b8139def41639f959d0e96260c82, State: Initialized, Role: FOLLOWER
I20260620 18:34:33.843008 14275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } }
Jun 20 18:34:33 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:33.843089 14275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:33.843133 14275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:33.843238 14275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:33.846988 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:33.836082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.10.109.126:42263)
0620 18:34:33.836247 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:33.836254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:33.836279 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:33.837264 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:33.837270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:33.837315 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:33.837431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:33.838250 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.839277 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.840281 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:33.840726 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:33.840809 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:33.843649 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:33.843655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:33.843660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:33.843670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:33.845358 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:33.845740 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:33.845743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:33.845744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:33.845829 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:33.846039 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:33.846040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:33.846041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:33.846269 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:33.846422 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:33.846560 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:34:33.846691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 18:34:33.897696 14275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } }
I20260620 18:34:33.897894 14275 leader_election.cc:302] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0201b8139def41639f959d0e96260c82; no voters:
I20260620 18:34:33.898128 14275 leader_election.cc:288] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:33.898203 14280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:33.898432 14275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:33.898452 14280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [term 1 LEADER]: Becoming Leader. State: Replica: 0201b8139def41639f959d0e96260c82, State: Running, Role: LEADER
I20260620 18:34:33.898713 14280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } }
I20260620 18:34:33.899123 14281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0201b8139def41639f959d0e96260c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } } }
I20260620 18:34:33.899214 14281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:33.899704 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0201b8139def41639f959d0e96260c82. Latest consensus state: current_term: 1 leader_uuid: "0201b8139def41639f959d0e96260c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201b8139def41639f959d0e96260c82" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42263 } } }
I20260620 18:34:33.900010 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:33.900166 14289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:33.900637 14289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:33.902336 14289 catalog_manager.cc:1347] Generated new cluster ID: 7f19fcb4a43245e2a81d2d7cc580fbab
I20260620 18:34:33.902395 14289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:33.918532 14289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:33.919138 14289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:33.922758 14289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0201b8139def41639f959d0e96260c82: Generated new TSK 0
I20260620 18:34:33.922955 14289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
W20260620 18:34:34.803323 10677 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.109.65: real 0.867s user 0.001s sys 0.002s
I20260620 18:34:34.803629 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:34.883575 14301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:34.883762 14301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:34.883785 14301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:34.884922 14301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:34.884966 14301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:34.884991 14301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:34.886451 14301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:34.886715 14301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:34.886989 14301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:34.891991 14309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:34.892050 14306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:34.891987 14307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:34.892021 14301 server_base.cc:1034] running on GCE node
I20260620 18:34:34.892380 14301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:34.892627 14301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:34.893839 14301 hybrid_clock.cc:648] HybridClock initialized: now 1781980474893814 us; error 35 us; skew 500 ppm
Jun 20 18:34:34 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980474, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:34.900068 14301 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:34.900583 14301 webserver.cc:460] Webserver started at http://127.10.109.65:39533/ using document root <none> and password file <none>
I20260620 18:34:34.900830 14301 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:34.900898 14301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:34.901036 14301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:34.902073 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "47ecc054a97447f485ae9923f6951ab7"
format_stamp: "Formatted at 2026-06-20 18:34:34 on dist-test-slave-3jql"
server_key: "e85b0a6a0ad284e0a69a1f73a11bfbe0"
server_key_iv: "322824086c6871836e04f7219a92ca79"
server_key_version: "encryptionkey@0"
I20260620 18:34:34.902438 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "47ecc054a97447f485ae9923f6951ab7"
format_stamp: "Formatted at 2026-06-20 18:34:34 on dist-test-slave-3jql"
server_key: "e85b0a6a0ad284e0a69a1f73a11bfbe0"
server_key_iv: "322824086c6871836e04f7219a92ca79"
server_key_version: "encryptionkey@0"
I20260620 18:34:34.903843 14301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:34.904597 14316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:34.904786 14301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:34.904874 14301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "47ecc054a97447f485ae9923f6951ab7"
format_stamp: "Formatted at 2026-06-20 18:34:34 on dist-test-slave-3jql"
server_key: "e85b0a6a0ad284e0a69a1f73a11bfbe0"
server_key_iv: "322824086c6871836e04f7219a92ca79"
server_key_version: "encryptionkey@0"
I20260620 18:34:34.904958 14301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:34.919544 14301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:34.922049 14301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:34.922165 14301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:34.922358 14301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:34.922596 14301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:34.922626 14301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:34.922648 14301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:34.922660 14301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:34.928812 14301 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:45277
I20260620 18:34:34.928889 14429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:45277 every 8 connection(s)
I20260620 18:34:34.929203 14301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:34 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980474, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:34.938369 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14301
I20260620 18:34:34.938493 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:34.938683 10677 external_mini_cluster.cc:1428] Setting key c271204020f8aeca8cb035598b31d1ca
I20260620 18:34:34.939950 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.929967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:45683 (local address 127.10.109.126:42263)
0620 18:34:34.930902 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:34.930905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:34.930919 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:34:34.931264 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:34.931265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:34.931279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:34.931350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:34.931951 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.932458 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.933467 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.933679 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.933791 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:34.936462 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:34.936463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:34.936464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:34.936470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:34.938355 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:34.938864 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:34.938865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:34.938866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:34.938933 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:34.939242 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:34.939242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:34.939242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:34.939396 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:34.939470 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:34.939764 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:34:34.939855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":72,"threads_started":1}
I20260620 18:34:34.940135 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:34.929940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42263 (local address 127.10.109.65:45683)
0620 18:34:34.930106 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:34.930141 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:34.931154 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:34.931421 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:34.931429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:34.931666 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:34.931846 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:34.931850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.932563 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.932564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:34.933369 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:34.933371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:34.933800 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:34.933801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:34.933923 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:34.934468 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:34.934478 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:34.936355 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:34.938455 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:34.938468 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:34.938470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:34.938789 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:34.939028 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:34.939029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:34.939029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:34.939167 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:34.939500 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:34.939506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:34.939630 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:34:34.939729 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:34:34.939875 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260620 18:34:34.940593 14430 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42263
I20260620 18:34:34.940704 14430 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:34.940861 14430 heartbeater.cc:511] Master 127.10.109.126:42263 requested a full tablet report, sending...
I20260620 18:34:34.941277 14239 ts_manager.cc:194] Registered new tserver with Master: 47ecc054a97447f485ae9923f6951ab7 (127.10.109.65:45277)
I20260620 18:34:34.941859 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:45683
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:34.966612 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:35.052217 14436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:35.052412 14436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:35.052433 14436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:35.053504 14436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:35.053539 14436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:35.053565 14436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:35.055037 14436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:35.055217 14436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:35.055385 14436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:35.060214 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:35.060228 14444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:35.060228 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:35.060542 14436 server_base.cc:1034] running on GCE node
I20260620 18:34:35.060763 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:35.061014 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:35.062172 14436 hybrid_clock.cc:648] HybridClock initialized: now 1781980475062159 us; error 25 us; skew 500 ppm
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:35.064646 14436 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:35.065099 14436 webserver.cc:460] Webserver started at http://127.10.109.66:46049/ using document root <none> and password file <none>
I20260620 18:34:35.065320 14436 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:35.065366 14436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:35.065512 14436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:35.066658 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "b5c2f88bfaf940c69c19dca16a2286b9"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "0df6e7e67016c7c73f6560dcb10a0aca"
server_key_iv: "618441e354e19303d198533d11474377"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.067102 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "b5c2f88bfaf940c69c19dca16a2286b9"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "0df6e7e67016c7c73f6560dcb10a0aca"
server_key_iv: "618441e354e19303d198533d11474377"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.068405 14436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:35.069196 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.069383 14436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:35.069461 14436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "b5c2f88bfaf940c69c19dca16a2286b9"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "0df6e7e67016c7c73f6560dcb10a0aca"
server_key_iv: "618441e354e19303d198533d11474377"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.069521 14436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:35.084156 14436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:35.086410 14436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:35.086553 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:35.086810 14436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:35.087117 14436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:35.087152 14436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.087173 14436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:35.087186 14436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.092840 14436 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:42903
I20260620 18:34:35.092893 14564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:42903 every 8 connection(s)
I20260620 18:34:35.093214 14436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:35.101565 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14436
I20260620 18:34:35.101691 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:35.101889 10677 external_mini_cluster.cc:1428] Setting key 27dccdcc5a3ceded154f4af69b2020e0
I20260620 18:34:35.103539 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.093899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42263 (local address 127.10.109.66:56861)
0620 18:34:35.094098 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:35.094131 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:35.095126 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:35.095392 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:35.095395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:35.095647 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:35.095830 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:35.095833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.096541 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.096542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:35.097327 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:35.097329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.097737 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.097738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:35.097861 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.098377 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:35.098385 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:35.099985 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:35.102184 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:35.102191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:35.102193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:35.102406 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:35.102636 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:35.102637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:35.102637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:35.102793 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:35.103074 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:35.103079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:35.103190 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 18:34:35.103267 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:34:35.103380 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260620 18:34:35.103585 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.093936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:56861 (local address 127.10.109.126:42263)
0620 18:34:35.094002 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:35.094004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:35.094218 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:34:35.095247 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:35.095249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:35.095264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:35.095321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:35.095930 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.096438 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.097434 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.097643 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.097729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.100112 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:35.100113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:35.100114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:35.100121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:35.102077 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.102502 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.102504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.102504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.102572 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.102854 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.102855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.102855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.103004 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:35.103078 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:35.103324 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:34:35.103431 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:34:35.103945 14565 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42263
I20260620 18:34:35.104054 14565 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:35.104223 14565 heartbeater.cc:511] Master 127.10.109.126:42263 requested a full tablet report, sending...
I20260620 18:34:35.104581 14239 ts_manager.cc:194] Registered new tserver with Master: b5c2f88bfaf940c69c19dca16a2286b9 (127.10.109.66:42903)
I20260620 18:34:35.105093 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:56861
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:35.130292 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:35.211022 14570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:35.211200 14570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:35.211222 14570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:35.212342 14570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:35.212380 14570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:35.212438 14570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:35.213894 14570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:35.214128 14570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:35.214388 14570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:35.219524 14575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:35.219611 14576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:35.219550 14578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:35.219619 14570 server_base.cc:1034] running on GCE node
I20260620 18:34:35.219836 14570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:35.220081 14570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:35.221240 14570 hybrid_clock.cc:648] HybridClock initialized: now 1781980475221225 us; error 27 us; skew 500 ppm
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:35.223678 14570 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:35.224118 14570 webserver.cc:460] Webserver started at http://127.10.109.67:33943/ using document root <none> and password file <none>
I20260620 18:34:35.224352 14570 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:35.224429 14570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:35.224550 14570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:35.225478 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "88ee69c7209d4a368b55037d70de2884"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "7edf35fe53a34da0478d500c705c4aa5"
server_key_iv: "ad9949bd11ab76a411f87389cd617197"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.225767 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "88ee69c7209d4a368b55037d70de2884"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "7edf35fe53a34da0478d500c705c4aa5"
server_key_iv: "ad9949bd11ab76a411f87389cd617197"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.227058 14570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:35.227754 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.227923 14570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:35.227994 14570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "88ee69c7209d4a368b55037d70de2884"
format_stamp: "Formatted at 2026-06-20 18:34:35 on dist-test-slave-3jql"
server_key: "7edf35fe53a34da0478d500c705c4aa5"
server_key_iv: "ad9949bd11ab76a411f87389cd617197"
server_key_version: "encryptionkey@0"
I20260620 18:34:35.228044 14570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:35.258543 14570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:35.260831 14570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:35.260937 14570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:35.261113 14570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:35.261510 14570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:35.261545 14570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.261590 14570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:35.261617 14570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:35.267560 14570 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40623
I20260620 18:34:35.267620 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40623 every 8 connection(s)
I20260620 18:34:35.267923 14570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:35.275092 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14570
I20260620 18:34:35.275177 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:35.275403 10677 external_mini_cluster.cc:1428] Setting key 54f51fd47989678a6da77a265a76608f
I20260620 18:34:35.277500 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.268608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:44211 (local address 127.10.109.126:42263)
0620 18:34:35.268669 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:35.268671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:35.268879 (+ 208us) server_negotiation.cc:407] Connection header received
0620 18:34:35.269756 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:35.269756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:35.269766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:35.269829 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:35.270390 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.271122 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.272058 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.272328 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.272390 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.274497 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:35.274498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:35.274499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:35.274505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:35.276194 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.276616 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.276617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.276618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.276673 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.276907 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.276908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.276908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.277052 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:35.277115 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:35.277372 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:34:35.277437 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:35.277662 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.268598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42263 (local address 127.10.109.67:44211)
0620 18:34:35.268751 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:35.268791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:35.269671 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:35.269907 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:35.269910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:35.270145 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:35.270320 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:35.270323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.271214 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.271215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:35.271978 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:35.271980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.272401 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.272402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:35.272518 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.272991 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:35.273000 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:35.274398 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:35.276321 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:35.276326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:35.276327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:35.276532 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:35.276740 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:35.276741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:35.276741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:35.276828 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:35.277140 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:35.277145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:35.277253 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 18:34:35.277345 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:34:35.277455 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 18:34:35.278183 14699 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42263
I20260620 18:34:35.278294 14699 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:35.278460 14699 heartbeater.cc:511] Master 127.10.109.126:42263 requested a full tablet report, sending...
I20260620 18:34:35.278906 14239 ts_manager.cc:194] Registered new tserver with Master: 88ee69c7209d4a368b55037d70de2884 (127.10.109.67:40623)
I20260620 18:34:35.279381 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:44211
I20260620 18:34:35.287230 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:35.311321 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.302796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.10.109.65:45277)
0620 18:34:35.302994 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:35.302999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:35.303010 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:35.303038 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:35.303040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:35.303054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:35.303126 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:35.303776 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.304513 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.305422 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.305761 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.305828 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.308384 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:35.308391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:35.308392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:35.308399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:35.310000 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.310334 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.310336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.310337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.310407 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.310665 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.310666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.310666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.310890 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:35.311020 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:35.311174 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:34:35.311239 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:35.320498 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:35.312885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.10.109.126:42263)
0620 18:34:35.312957 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:35.312959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:35.312969 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:35.313074 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:35.313076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:35.313086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:35.313144 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:35.313613 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.314151 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.315178 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:35.315391 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:35.315477 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:35.317697 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:35.317699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:35.317699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:35.317708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:35.319202 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.319514 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.319514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.319515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.319564 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:35.319860 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:35.319862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:35.319862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:35.320012 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:35.320100 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:35.320239 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:34:35.320369 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 18:34:35.320811 14239 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114
I20260620 18:34:35.321192 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14301
I20260620 18:34:35.325925 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14436
I20260620 18:34:35.330360 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14570
I20260620 18:34:35.334661 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14207
2026-06-20T18:34:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5624 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14711](info): set up 2 sockets
Jun 20 18:34:35 dist-test-slave-3jql krb5kdc[14711](info): commencing operation
krb5kdc: starting...
W20260620 18:34:36.395361 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:36.514889 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41735
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:36.597580 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:36.597790 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:36.597827 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:36.599035 14724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:36.599092 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:36.599118 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:36.599146 14724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:36.599170 14724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:36.600607 14724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:36.600857 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:36.601087 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:36.605854 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.605898 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.605855 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:36.605881 14724 server_base.cc:1034] running on GCE node
I20260620 18:34:36.606348 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:36.606631 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:36.607795 14724 hybrid_clock.cc:648] HybridClock initialized: now 1781980476607777 us; error 48 us; skew 500 ppm
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:36.610255 14724 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:36.610622 14724 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:36.610692 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:36.610885 14724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:36.611886 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "128e0881fe3e480eb7a131dc657091db"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "265b2b35475a76024d7ccf729673d546"
server_key_iv: "ae7e5b67993965c443503e8c2484d1ad"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.612237 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "128e0881fe3e480eb7a131dc657091db"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "265b2b35475a76024d7ccf729673d546"
server_key_iv: "ae7e5b67993965c443503e8c2484d1ad"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.613487 14724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:36.614298 14738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.614502 14724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:36.614604 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "128e0881fe3e480eb7a131dc657091db"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "265b2b35475a76024d7ccf729673d546"
server_key_iv: "ae7e5b67993965c443503e8c2484d1ad"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.614696 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:36.648968 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:36.651150 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:36.651302 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:36.655267 14724 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:41735
I20260620 18:34:36.655329 14790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:41735 every 8 connection(s)
I20260620 18:34:36.655592 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:36.656558 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:36.658833 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: Bootstrap starting.
I20260620 18:34:36.659408 14791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:36.659631 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14724
I20260620 18:34:36.659745 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:36.659816 14791 log.cc:824] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:36.659933 10677 external_mini_cluster.cc:1428] Setting key 0c71011f6d705c286756e558bc59ff6c
I20260620 18:34:36.660574 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: No bootstrap required, opened a new log
I20260620 18:34:36.662659 14791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } }
I20260620 18:34:36.662868 14791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:36.662923 14791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128e0881fe3e480eb7a131dc657091db, State: Initialized, Role: FOLLOWER
I20260620 18:34:36.663046 14791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } }
I20260620 18:34:36.663121 14791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:36.663143 14791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:36.663208 14791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:36.663745 14791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } }
I20260620 18:34:36.663867 14791 leader_election.cc:302] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128e0881fe3e480eb7a131dc657091db; no voters:
I20260620 18:34:36.664043 14791 leader_election.cc:288] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:36.664068 14796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:36.664183 14796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [term 1 LEADER]: Becoming Leader. State: Replica: 128e0881fe3e480eb7a131dc657091db, State: Running, Role: LEADER
I20260620 18:34:36.664300 14791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:36.664321 14796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } }
I20260620 18:34:36.664834 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128e0881fe3e480eb7a131dc657091db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } } }
I20260620 18:34:36.664923 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:36.665176 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128e0881fe3e480eb7a131dc657091db. Latest consensus state: current_term: 1 leader_uuid: "128e0881fe3e480eb7a131dc657091db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128e0881fe3e480eb7a131dc657091db" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41735 } } }
I20260620 18:34:36.665298 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:36.665489 14805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:36.665990 14805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:36.667485 14805 catalog_manager.cc:1347] Generated new cluster ID: 403ead7ef95e4312813571480f676de2
I20260620 18:34:36.667537 14805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:36.674402 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:36.660379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.10.109.126:41735)
0620 18:34:36.660584 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:36.660593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:36.660623 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:34:36.661072 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:36.661078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:36.661121 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:36.661266 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:36.661868 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.662723 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.663695 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.664225 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.664577 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:36.667122 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:36.667126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:36.667126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:36.667138 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:36.672815 (+ 5677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.673229 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.673232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.673232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.673326 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.673542 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.673543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.673543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.673809 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:36.673948 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:36.674081 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:34:36.674198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260620 18:34:36.680827 14805 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:36.681320 14805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:36.692605 14805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: Generated new TSK 0
I20260620 18:34:36.692791 14805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:36.721921 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:36.800851 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:36.801064 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:36.801088 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:36.802361 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:36.802409 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:36.802438 14817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:36.803982 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:36.804368 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:36.804651 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:36.810086 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.810104 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.810086 14825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:36.810367 14817 server_base.cc:1034] running on GCE node
I20260620 18:34:36.810665 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:36.810979 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:36.812191 14817 hybrid_clock.cc:648] HybridClock initialized: now 1781980476812167 us; error 29 us; skew 500 ppm
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:36.814821 14817 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:36.815109 14817 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:36.815155 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:36.815243 14817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:36.816238 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "5526287e5e594b5c855129467f5fcbe9"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "1d1056082241fba13079d51ab942a119"
server_key_iv: "bb84f7d9c881db2cad7a4f1a6b30b805"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.816545 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "5526287e5e594b5c855129467f5fcbe9"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "1d1056082241fba13079d51ab942a119"
server_key_iv: "bb84f7d9c881db2cad7a4f1a6b30b805"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.818115 14817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:36.818986 14831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.819219 14817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:36.819293 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "5526287e5e594b5c855129467f5fcbe9"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "1d1056082241fba13079d51ab942a119"
server_key_iv: "bb84f7d9c881db2cad7a4f1a6b30b805"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.819387 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:36.835713 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:36.839092 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:36.839274 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:36.839550 14817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:36.839881 14817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:36.839916 14817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.839939 14817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:36.839984 14817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.848227 14817 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:42889
I20260620 18:34:36.848315 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:42889 every 8 connection(s)
I20260620 18:34:36.848594 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:36.856468 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14817
I20260620 18:34:36.856591 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:36.856781 10677 external_mini_cluster.cc:1428] Setting key 373a7c22086bd18b1a53ff3093688b33
I20260620 18:34:36.859913 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:36.849865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:44571 (local address 127.10.109.126:41735)
0620 18:34:36.849951 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:36.849953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:36.850199 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:34:36.850710 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:36.850711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:36.850722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:36.850874 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:36.851816 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.852370 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.853392 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.853590 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.853649 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:36.856047 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:36.856049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:36.856049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:36.856056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:36.857893 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.858414 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.858416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.858416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.858487 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:36.858894 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:36.858895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:36.858895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:36.859060 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:36.859155 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:36.859638 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 18:34:36.859741 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:34:36.860080 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:36.849792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41735 (local address 127.10.109.65:44571)
0620 18:34:36.850030 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:36.850069 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:36.850552 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:36.850936 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:36.850940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:36.851203 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:36.851704 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:36.851709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.852505 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.852506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:36.853292 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:36.853294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:36.853723 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:36.853724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:36.853839 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:36.854465 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:36.854473 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:36.855947 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:36.857980 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:36.857987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:36.857990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:36.858314 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:36.858562 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:36.858563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:36.858564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:36.858709 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:36.859216 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:36.859231 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:36.859434 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 18:34:36.859589 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 18:34:36.859782 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":111,"threads_started":1}
I20260620 18:34:36.860675 14945 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41735
I20260620 18:34:36.860824 14945 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:36.861020 14945 heartbeater.cc:511] Master 127.10.109.126:41735 requested a full tablet report, sending...
I20260620 18:34:36.861583 14755 ts_manager.cc:194] Registered new tserver with Master: 5526287e5e594b5c855129467f5fcbe9 (127.10.109.65:42889)
I20260620 18:34:36.862555 14755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:44571
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:36.888538 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:36.969225 14950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:36.969445 14950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:36.969480 14950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:36.970595 14950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:36.970662 14950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:36.970709 14950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:36.972172 14950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:36.972427 14950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:36.972643 14950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:36.977523 14958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:36.977675 14950 server_base.cc:1034] running on GCE node
W20260620 18:34:36.977797 14956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:36.977799 14955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:36.978089 14950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:36.978327 14950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:36.979485 14950 hybrid_clock.cc:648] HybridClock initialized: now 1781980476979456 us; error 50 us; skew 500 ppm
Jun 20 18:34:36 dist-test-slave-3jql krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:36.981905 14950 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:36.982177 14950 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:36.982221 14950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:36.982306 14950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:36.983351 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "febd2b7cb0cd481782e36cd668ce4cf2"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "d23c2140bdf9659961e92b360d3d29c5"
server_key_iv: "a2e22cf18fa979a10b859e2f1328e176"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.983637 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "febd2b7cb0cd481782e36cd668ce4cf2"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "d23c2140bdf9659961e92b360d3d29c5"
server_key_iv: "a2e22cf18fa979a10b859e2f1328e176"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.984839 14950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:36.985565 14964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.985805 14950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:36.985872 14950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "febd2b7cb0cd481782e36cd668ce4cf2"
format_stamp: "Formatted at 2026-06-20 18:34:36 on dist-test-slave-3jql"
server_key: "d23c2140bdf9659961e92b360d3d29c5"
server_key_iv: "a2e22cf18fa979a10b859e2f1328e176"
server_key_version: "encryptionkey@0"
I20260620 18:34:36.985915 14950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:36.993631 14950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:36.995797 14950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:36.995910 14950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:36.996084 14950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:36.996337 14950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:36.996366 14950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:36.996387 14950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:36.996399 14950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.002233 14950 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:34053
I20260620 18:34:37.002302 15077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:34053 every 8 connection(s)
I20260620 18:34:37.002532 14950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:37.003516 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 14950
I20260620 18:34:37.003629 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:37.003811 10677 external_mini_cluster.cc:1428] Setting key f8160b6a97d34fb34bc3011c271703ef
Jun 20 18:34:37 dist-test-slave-3jql krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:34:37.013211 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.003209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:57657 (local address 127.10.109.126:41735)
0620 18:34:37.003297 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.003299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.003551 (+ 252us) server_negotiation.cc:407] Connection header received
0620 18:34:37.004217 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.004218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.004230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.004305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.005519 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.006131 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.007288 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.007510 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.007609 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.010115 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.010117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.010118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:37.010140 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.011749 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.012219 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.012221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.012221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.012283 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.012568 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.012569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.012569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.012696 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.012775 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.013041 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.013118 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:37.013397 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41735 (local address 127.10.109.66:57657)
0620 18:34:37.003388 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.003435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.004105 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.004355 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.004358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.004735 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:37.005395 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.005401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.006256 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.006257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.007200 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.007203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.007621 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.007622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.007750 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.008290 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:37.008298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:37.010011 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:37.011853 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:37.011860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:37.011862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:37.012106 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:37.012353 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:37.012354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:37.012354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:37.012495 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:37.012764 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:37.012771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:37.012909 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.013020 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.013172 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260620 18:34:37.013909 15078 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41735
I20260620 18:34:37.013993 15078 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:37.014098 15078 heartbeater.cc:511] Master 127.10.109.126:41735 requested a full tablet report, sending...
I20260620 18:34:37.014436 14755 ts_manager.cc:194] Registered new tserver with Master: febd2b7cb0cd481782e36cd668ce4cf2 (127.10.109.66:34053)
I20260620 18:34:37.014912 14755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:57657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:37.030174 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:37.108525 15083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:37.108707 15083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:37.108758 15083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:37.109969 15083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:37.110009 15083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:37.110036 15083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:37.111531 15083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:37.111745 15083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:37.111970 15083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:37.116747 15091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.116801 15089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.116748 15088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.117010 15083 server_base.cc:1034] running on GCE node
I20260620 18:34:37.117214 15083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:37.117440 15083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:37.118610 15083 hybrid_clock.cc:648] HybridClock initialized: now 1781980477118609 us; error 51 us; skew 500 ppm
Jun 20 18:34:37 dist-test-slave-3jql krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:37.120995 15083 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:37.121311 15083 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:37.121397 15083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:37.121534 15083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:37.122556 15083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "97d047a7851443b995043519e92712ee"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "1df68d744e171de74750c4c55bb5abd2"
server_key_iv: "09e5d907d03ac3ea4974036ec394fb10"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.122933 15083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "97d047a7851443b995043519e92712ee"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "1df68d744e171de74750c4c55bb5abd2"
server_key_iv: "09e5d907d03ac3ea4974036ec394fb10"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.124219 15083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:37.125095 15097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.125332 15083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:37.125432 15083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "97d047a7851443b995043519e92712ee"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "1df68d744e171de74750c4c55bb5abd2"
server_key_iv: "09e5d907d03ac3ea4974036ec394fb10"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.125516 15083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:37.142648 15083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:37.144606 15083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:37.144765 15083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:37.145000 15083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:37.145311 15083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:37.145366 15083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.145406 15083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:37.145460 15083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.150995 15083 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:34393
I20260620 18:34:37.151055 15210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:34393 every 8 connection(s)
I20260620 18:34:37.151320 15083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:37.154428 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15083
I20260620 18:34:37.154546 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:37.154731 10677 external_mini_cluster.cc:1428] Setting key 37dca75e643d37cd6d7aeeef719f81f8
Jun 20 18:34:37 dist-test-slave-3jql krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:37.161163 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.152036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:42761 (local address 127.10.109.126:41735)
0620 18:34:37.152079 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.152082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.152254 (+ 172us) server_negotiation.cc:407] Connection header received
0620 18:34:37.152777 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.152778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.152790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.152854 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.153750 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.154298 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.155767 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.155979 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.156038 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.158116 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.158117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.158117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:37.158124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.159614 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.160094 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.160096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.160097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.160228 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.160605 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.160607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.160608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.160737 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.160816 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.161027 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.161100 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:34:37.161456 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.151986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41735 (local address 127.10.109.67:42761)
0620 18:34:37.152131 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.152173 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.152675 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.152920 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.152924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.153172 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:37.153646 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.153650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.154398 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.154400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.155688 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.155691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.156089 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.156091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.156197 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.156658 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:37.156666 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:37.158034 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:37.159692 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:37.159698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:37.159699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:37.159900 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:37.160402 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:37.160402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:37.160403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:37.160505 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:37.160812 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:37.160816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:37.160921 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.161008 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.161122 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260620 18:34:37.162060 15211 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41735
I20260620 18:34:37.162154 15211 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:37.162261 15211 heartbeater.cc:511] Master 127.10.109.126:41735 requested a full tablet report, sending...
I20260620 18:34:37.162658 14755 ts_manager.cc:194] Registered new tserver with Master: 97d047a7851443b995043519e92712ee (127.10.109.67:34393)
I20260620 18:34:37.163087 14755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:42761
I20260620 18:34:37.166510 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:37.173964 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.167881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.10.109.126:41735)
0620 18:34:37.167937 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.167939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.168135 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:34:37.168208 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.168209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.168219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.168272 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.168843 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.169301 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.170027 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.170214 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.170295 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.171456 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.171457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.171458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:37.171464 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.172815 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.173180 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.173181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.173181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.173239 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:37.173434 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:37.173435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:37.173436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:37.173576 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.173681 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.173793 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.173869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:37.175611 14755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:37.177157 14755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:37.183072 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42889 (local address 127.0.0.1:48926)
0620 18:34:37.179257 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.179285 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.179366 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.179772 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.179775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.179777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:37.179883 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.179885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.180830 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.180832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.181915 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.181917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.182769 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.182770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.182867 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.182882 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.182952 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.183003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":66,"threads_started":1}
I20260620 18:34:37.183087 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.10.109.65:42889)
0620 18:34:37.179529 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.179533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.179543 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:37.179583 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.179586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.179599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.179677 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:37.179969 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.180736 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.182006 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.182672 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.182786 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.182846 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.182975 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.183012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260620 18:34:37.183470 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34393 (local address 127.0.0.1:45736)
0620 18:34:37.179665 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.179675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.179713 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.180041 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.180042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.180043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:37.180077 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.180078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.181319 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.181320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.182246 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.182248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.183222 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.183223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.183296 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.183301 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.183348 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.183389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":25,"thread_start_us":41,"threads_started":1}
I20260620 18:34:37.183588 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:34053 (local address 127.0.0.1:48074)
0620 18:34:37.179642 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.179653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.179710 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.179974 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.179975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.179977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:37.180032 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.180034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.181308 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.181309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.182421 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.182423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.183314 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.183315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.183380 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.183384 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.183447 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.183488 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":26,"thread_start_us":56,"threads_started":1}
I20260620 18:34:37.183595 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.10.109.66:34053)
0620 18:34:37.179624 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.179629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.179770 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:34:37.179808 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.179811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.179824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.179892 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:37.180111 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.181199 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.182503 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.183240 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.183307 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.183359 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.183458 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.183502 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260620 18:34:37.183761 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.179595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.10.109.67:34393)
0620 18:34:37.179761 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.179764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.179773 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:37.179802 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.179803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.179816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.179897 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:37.180153 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.181203 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.182327 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.183069 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.183138 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.183184 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.183638 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.183686 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260620 18:34:37.184017 14879 tablet_service.cc:1467] Processing CreateTablet for tablet fbf941e63c1145ffb8fab55e907e39bc (DEFAULT_TABLE table=test-table [id=d7b6e36db6ae47a4ba6b190ddcf3a3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:37.184198 15012 tablet_service.cc:1467] Processing CreateTablet for tablet fbf941e63c1145ffb8fab55e907e39bc (DEFAULT_TABLE table=test-table [id=d7b6e36db6ae47a4ba6b190ddcf3a3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:37.184378 14879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf941e63c1145ffb8fab55e907e39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:37.184371 15145 tablet_service.cc:1467] Processing CreateTablet for tablet fbf941e63c1145ffb8fab55e907e39bc (DEFAULT_TABLE table=test-table [id=d7b6e36db6ae47a4ba6b190ddcf3a3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:37.184501 15012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf941e63c1145ffb8fab55e907e39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:37.184644 15145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf941e63c1145ffb8fab55e907e39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:37.186718 15225 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Bootstrap starting.
I20260620 18:34:37.187692 15226 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Bootstrap starting.
I20260620 18:34:37.187805 15225 tablet_bootstrap.cc:654] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:37.187911 15227 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Bootstrap starting.
I20260620 18:34:37.188405 15226 tablet_bootstrap.cc:654] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:37.188443 15225 log.cc:824] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:37.188504 15227 tablet_bootstrap.cc:654] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:37.188984 15226 log.cc:824] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:37.189170 15227 log.cc:824] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:37.189246 15225 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: No bootstrap required, opened a new log
I20260620 18:34:37.189311 15225 ts_tablet_manager.cc:1397] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:34:37.189872 15226 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: No bootstrap required, opened a new log
I20260620 18:34:37.189936 15226 ts_tablet_manager.cc:1397] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:34:37.191166 15227 tablet_bootstrap.cc:492] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: No bootstrap required, opened a new log
I20260620 18:34:37.191247 15227 ts_tablet_manager.cc:1397] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:34:37.191421 15225 raft_consensus.cc:348] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.191607 15225 raft_consensus.cc:374] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:37.191645 15225 raft_consensus.cc:729] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5526287e5e594b5c855129467f5fcbe9, State: Initialized, Role: FOLLOWER
I20260620 18:34:37.191776 15225 consensus_queue.cc:260] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.191831 15226 raft_consensus.cc:348] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.191979 15226 raft_consensus.cc:374] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:37.192021 15226 raft_consensus.cc:729] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d047a7851443b995043519e92712ee, State: Initialized, Role: FOLLOWER
I20260620 18:34:37.192060 15225 ts_tablet_manager.cc:1428] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:34:37.192127 15226 consensus_queue.cc:260] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.192179 14945 heartbeater.cc:503] Master 127.10.109.126:41735 was elected leader, sending a full tablet report...
I20260620 18:34:37.192428 15226 ts_tablet_manager.cc:1428] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:34:37.192543 15211 heartbeater.cc:503] Master 127.10.109.126:41735 was elected leader, sending a full tablet report...
I20260620 18:34:37.193225 15227 raft_consensus.cc:348] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.193379 15227 raft_consensus.cc:374] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:37.193423 15227 raft_consensus.cc:729] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febd2b7cb0cd481782e36cd668ce4cf2, State: Initialized, Role: FOLLOWER
I20260620 18:34:37.193531 15227 consensus_queue.cc:260] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.193782 15227 ts_tablet_manager.cc:1428] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:34:37.193804 15078 heartbeater.cc:503] Master 127.10.109.126:41735 was elected leader, sending a full tablet report...
W20260620 18:34:37.253086 15079 tablet.cc:2369] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:37.349455 14946 tablet.cc:2369] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:37.401959 15212 tablet.cc:2369] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:37.499078 15233 raft_consensus.cc:482] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:37.499215 15233 raft_consensus.cc:504] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.499725 15233 leader_election.cc:288] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5526287e5e594b5c855129467f5fcbe9 (127.10.109.65:42889), 97d047a7851443b995043519e92712ee (127.10.109.67:34393)
I20260620 18:34:37.504124 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.499902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42889 (local address 127.10.109.66:60637)
0620 18:34:37.499992 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.500006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.500055 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.500304 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.500305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.500307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:37.500380 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.500381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.501434 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.501435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.502863 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.502866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.503814 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.503909 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.503913 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.503988 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.504031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":88}
I20260620 18:34:37.504166 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.500070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:60637 (local address 127.10.109.65:42889)
0620 18:34:37.500154 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.500156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.500167 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:37.500200 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.500201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.500213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.500278 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:37.500475 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.501326 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503023 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.503655 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503748 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.503756 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.504047 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.504093 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:34:37.504166 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.500148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:38727 (local address 127.10.109.67:34393)
0620 18:34:37.500222 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.500224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.500234 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:37.500358 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.500359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.500367 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.500414 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:37.500561 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.501343 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503023 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.503655 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.503756 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.504047 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.504093 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:34:37.504130 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.500032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34393 (local address 127.10.109.66:38727)
0620 18:34:37.500149 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.500161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.500230 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.500431 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.500432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.500434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:37.500495 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.500497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.501434 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.501435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.502863 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:37.502866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.503814 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.503815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:37.503939 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.503943 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.503999 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.504040 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":114,"thread_start_us":65,"threads_started":1}
I20260620 18:34:37.504469 14899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf941e63c1145ffb8fab55e907e39bc" candidate_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5526287e5e594b5c855129467f5fcbe9" is_pre_election: true
I20260620 18:34:37.504624 14899 raft_consensus.cc:2454] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febd2b7cb0cd481782e36cd668ce4cf2 in term 0.
I20260620 18:34:37.504552 15165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf941e63c1145ffb8fab55e907e39bc" candidate_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d047a7851443b995043519e92712ee" is_pre_election: true
I20260620 18:34:37.504739 15165 raft_consensus.cc:2454] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febd2b7cb0cd481782e36cd668ce4cf2 in term 0.
I20260620 18:34:37.504904 14967 leader_election.cc:302] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5526287e5e594b5c855129467f5fcbe9, febd2b7cb0cd481782e36cd668ce4cf2; no voters:
I20260620 18:34:37.505021 15233 raft_consensus.cc:2781] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:37.505075 15233 raft_consensus.cc:482] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:37.505100 15233 raft_consensus.cc:3037] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:37.505769 15233 raft_consensus.cc:504] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.505903 15233 leader_election.cc:288] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [CANDIDATE]: Term 1 election: Requested vote from peers 5526287e5e594b5c855129467f5fcbe9 (127.10.109.65:42889), 97d047a7851443b995043519e92712ee (127.10.109.67:34393)
I20260620 18:34:37.506070 15165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf941e63c1145ffb8fab55e907e39bc" candidate_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d047a7851443b995043519e92712ee"
I20260620 18:34:37.506108 14899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf941e63c1145ffb8fab55e907e39bc" candidate_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5526287e5e594b5c855129467f5fcbe9"
I20260620 18:34:37.506172 14899 raft_consensus.cc:3037] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:37.506165 15165 raft_consensus.cc:3037] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:37.507053 15165 raft_consensus.cc:2454] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febd2b7cb0cd481782e36cd668ce4cf2 in term 1.
I20260620 18:34:37.507053 14899 raft_consensus.cc:2454] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febd2b7cb0cd481782e36cd668ce4cf2 in term 1.
I20260620 18:34:37.507262 14967 leader_election.cc:302] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5526287e5e594b5c855129467f5fcbe9, febd2b7cb0cd481782e36cd668ce4cf2; no voters:
I20260620 18:34:37.507362 15233 raft_consensus.cc:2781] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:37.507508 15233 raft_consensus.cc:686] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 1 LEADER]: Becoming Leader. State: Replica: febd2b7cb0cd481782e36cd668ce4cf2, State: Running, Role: LEADER
I20260620 18:34:37.507604 15233 consensus_queue.cc:237] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } }
I20260620 18:34:37.508471 14754 catalog_manager.cc:5495] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to febd2b7cb0cd481782e36cd668ce4cf2 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "febd2b7cb0cd481782e36cd668ce4cf2" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:37.535405 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.533075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.10.109.66:34053)
0620 18:34:37.533141 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.533143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.533150 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:34:37.533178 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.533179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.533193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.533251 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:37.533391 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.533859 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.534641 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:37.534861 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:37.534913 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:37.535071 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:37.535162 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:37.535222 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.535264 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.535281 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:37.537086 15165 raft_consensus.cc:1261] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 1 FOLLOWER]: Refusing update from remote peer febd2b7cb0cd481782e36cd668ce4cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:37.537096 14899 raft_consensus.cc:1261] T fbf941e63c1145ffb8fab55e907e39bc P 5526287e5e594b5c855129467f5fcbe9 [term 1 FOLLOWER]: Refusing update from remote peer febd2b7cb0cd481782e36cd668ce4cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:37.537515 15233 consensus_queue.cc:1035] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5526287e5e594b5c855129467f5fcbe9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:37.537681 15235 consensus_queue.cc:1035] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d047a7851443b995043519e92712ee" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:37.539985 15238 mvcc.cc:205] Tried to move back new op lower bound from 7298992035989450752 to 7298992035871944704. Current Snapshot: MvccSnapshot[applied={T|T < 7298992035989450752}]
I20260620 18:34:37.540380 15239 mvcc.cc:205] Tried to move back new op lower bound from 7298992035989450752 to 7298992035871944704. Current Snapshot: MvccSnapshot[applied={T|T < 7298992035989450752}]
I20260620 18:34:37.540514 15240 mvcc.cc:205] Tried to move back new op lower bound from 7298992035989450752 to 7298992035871944704. Current Snapshot: MvccSnapshot[applied={T|T < 7298992035989450752 or (T in {7298992035989450752})}]
I20260620 18:34:37.542822 14754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:37.542933 14754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:37.543773 14754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 128e0881fe3e480eb7a131dc657091db: Sending DeleteTablet for 3 replicas of tablet fbf941e63c1145ffb8fab55e907e39bc
I20260620 18:34:37.544178 14879 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf941e63c1145ffb8fab55e907e39bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:37 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:48926
I20260620 18:34:37.544268 15012 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf941e63c1145ffb8fab55e907e39bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:37 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:48074
I20260620 18:34:37.544399 15145 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf941e63c1145ffb8fab55e907e39bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:37 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:45736
I20260620 18:34:37.544482 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14817
I20260620 18:34:37.544581 15250 tablet_replica.cc:330] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: stopping tablet replica
I20260620 18:34:37.544673 15250 raft_consensus.cc:2229] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:37.544764 15250 raft_consensus.cc:2258] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:37.545080 15249 tablet_replica.cc:330] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: stopping tablet replica
I20260620 18:34:37.545177 15249 raft_consensus.cc:2229] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:37.545295 15249 raft_consensus.cc:2258] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:37.545583 15250 ts_tablet_manager.cc:1905] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:37.546945 15250 ts_tablet_manager.cc:1918] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:37.547067 15250 log.cc:1196] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/fbf941e63c1145ffb8fab55e907e39bc
I20260620 18:34:37.547107 15249 ts_tablet_manager.cc:1905] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:37.547447 15250 ts_tablet_manager.cc:1939] T fbf941e63c1145ffb8fab55e907e39bc P 97d047a7851443b995043519e92712ee: Deleting consensus metadata
I20260620 18:34:37.547883 14741 catalog_manager.cc:4841] TS 97d047a7851443b995043519e92712ee (127.10.109.67:34393): tablet fbf941e63c1145ffb8fab55e907e39bc (table test-table [id=d7b6e36db6ae47a4ba6b190ddcf3a3e6]) successfully deleted
I20260620 18:34:37.548529 15249 ts_tablet_manager.cc:1918] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:37.548619 15249 log.cc:1196] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/fbf941e63c1145ffb8fab55e907e39bc
I20260620 18:34:37.548831 15249 ts_tablet_manager.cc:1939] T fbf941e63c1145ffb8fab55e907e39bc P febd2b7cb0cd481782e36cd668ce4cf2: Deleting consensus metadata
I20260620 18:34:37.549197 14741 catalog_manager.cc:4841] TS febd2b7cb0cd481782e36cd668ce4cf2 (127.10.109.66:34053): tablet fbf941e63c1145ffb8fab55e907e39bc (table test-table [id=d7b6e36db6ae47a4ba6b190ddcf3a3e6]) successfully deleted
W20260620 18:34:37.553260 14741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:42889 (error 108)
I20260620 18:34:37.553510 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14950
I20260620 18:34:37.553740 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.553628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42889 (local address 127.0.0.1:48932)
0620 18:34:37.553673 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.553704 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 18:34:37.553836 14741 catalog_manager.cc:4584] TS 5526287e5e594b5c855129467f5fcbe9 (127.10.109.65:42889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbf941e63c1145ffb8fab55e907e39bc: Network error: Client connection negotiation failed: client connection to 127.10.109.65:42889: connect: Connection refused (error 111)
I20260620 18:34:37.558142 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15083
I20260620 18:34:37.562453 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 14724
2026-06-20T18:34:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2228 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:37Z Disabled control of system clock
I20260620 18:34:37.590277 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46431
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:34:37.670622 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:37.670890 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:37.670918 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:37.672060 15255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:37.672101 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:37.672114 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:37.672127 15255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:37.672178 15255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:37.673477 15255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:46431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:37.673715 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:37.673996 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:37.678637 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.678601 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.678611 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.679054 15255 server_base.cc:1034] running on GCE node
I20260620 18:34:37.679311 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:37.679610 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:37.680749 15255 hybrid_clock.cc:648] HybridClock initialized: now 1781980477680727 us; error 36 us; skew 500 ppm
I20260620 18:34:37.682040 15255 webserver.cc:460] Webserver started at http://127.10.109.126:46073/ using document root <none> and password file <none>
I20260620 18:34:37.682277 15255 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:37.682348 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:37.682504 15255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:37.683581 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "c78be1e0293a46d8bf582d4aaaecb129"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bb2585fbcdc4017380dbca317f0ca73b"
server_key_iv: "8e1c888de40aad9bc7cf26615d043667"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.683964 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "c78be1e0293a46d8bf582d4aaaecb129"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bb2585fbcdc4017380dbca317f0ca73b"
server_key_iv: "8e1c888de40aad9bc7cf26615d043667"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.685211 15255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:37.686014 15269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.686213 15255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:37.686308 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "c78be1e0293a46d8bf582d4aaaecb129"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bb2585fbcdc4017380dbca317f0ca73b"
server_key_iv: "8e1c888de40aad9bc7cf26615d043667"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.686395 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:37.689673 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:37.689920 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:37.690068 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:37.694063 15255 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:46431
I20260620 18:34:37.694087 15321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:46431 every 8 connection(s)
I20260620 18:34:37.694402 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:37.695082 15322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:37.697536 15322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: Bootstrap starting.
I20260620 18:34:37.698026 15322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:37.698249 15322 log.cc:824] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:37.698932 15322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: No bootstrap required, opened a new log
I20260620 18:34:37.700436 15322 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } }
I20260620 18:34:37.700560 15322 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:37.700582 15322 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78be1e0293a46d8bf582d4aaaecb129, State: Initialized, Role: FOLLOWER
I20260620 18:34:37.700672 15322 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } }
I20260620 18:34:37.700721 15322 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:37.700768 15322 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:37.700814 15322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:37.701311 15322 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } }
I20260620 18:34:37.701416 15322 leader_election.cc:302] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78be1e0293a46d8bf582d4aaaecb129; no voters:
I20260620 18:34:37.701589 15322 leader_election.cc:288] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:37.701624 15325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:37.701807 15325 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [term 1 LEADER]: Becoming Leader. State: Replica: c78be1e0293a46d8bf582d4aaaecb129, State: Running, Role: LEADER
I20260620 18:34:37.701840 15322 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:37.701901 15325 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } }
I20260620 18:34:37.702322 15327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78be1e0293a46d8bf582d4aaaecb129. Latest consensus state: current_term: 1 leader_uuid: "c78be1e0293a46d8bf582d4aaaecb129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } } }
I20260620 18:34:37.702412 15327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:37.702520 15326 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78be1e0293a46d8bf582d4aaaecb129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be1e0293a46d8bf582d4aaaecb129" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46431 } } }
I20260620 18:34:37.702591 15326 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:37.702652 15334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:37.703161 15334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:37.704291 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15255
I20260620 18:34:37.704393 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:37.704574 10677 external_mini_cluster.cc:1428] Setting key 910fafd1e7ee2b59aaf1e01b55268d11
I20260620 18:34:37.705230 15334 catalog_manager.cc:1347] Generated new cluster ID: 802181a98b724e1abed9331c8b749ac1
I20260620 18:34:37.705281 15334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:37.708300 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.705092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.10.109.126:46431)
0620 18:34:37.706897 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.706906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.706938 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:34:37.707048 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.707055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.707097 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.707235 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.707874 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.707878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.707878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:37.707883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.707929 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:37.707943 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.708012 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.708077 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.708132 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":73,"threads_started":1}
I20260620 18:34:37.719645 15334 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:37.719805 15334 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:37.725672 15334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: Generated new TSK 0
I20260620 18:34:37.726361 15334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:37.728876 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--builtin_ntp_servers=127.10.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:34:37.820595 15346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:37.820796 15346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:37.820818 15346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:37.821981 15346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:37.822021 15346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:37.822045 15346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:37.823467 15346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:37.823697 15346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:37.823998 15346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:37.829113 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.829270 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.829275 15346 server_base.cc:1034] running on GCE node
W20260620 18:34:37.829516 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.829754 15346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:37.830049 15346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:37.831230 15346 hybrid_clock.cc:648] HybridClock initialized: now 1781980477831209 us; error 35 us; skew 500 ppm
I20260620 18:34:37.832530 15346 webserver.cc:460] Webserver started at http://127.10.109.65:43533/ using document root <none> and password file <none>
I20260620 18:34:37.832763 15346 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:37.832808 15346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:37.832947 15346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:37.833964 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "932b3870f2e647918d2f71dc8e227d78"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "08dabdfe649620ea02d4dc7ac5157469"
server_key_iv: "218b3bdba680794deb84ee67c221bcf1"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.834326 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "932b3870f2e647918d2f71dc8e227d78"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "08dabdfe649620ea02d4dc7ac5157469"
server_key_iv: "218b3bdba680794deb84ee67c221bcf1"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.835703 15346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:37.836462 15360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.836678 15346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:37.836781 15346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "932b3870f2e647918d2f71dc8e227d78"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "08dabdfe649620ea02d4dc7ac5157469"
server_key_iv: "218b3bdba680794deb84ee67c221bcf1"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.836851 15346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:37.841477 15346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:37.841758 15346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:37.841890 15346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:37.842098 15346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:37.842413 15346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:37.842453 15346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.842482 15346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:37.842509 15346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.849879 15346 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:37959
I20260620 18:34:37.849908 15473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:37959 every 8 connection(s)
I20260620 18:34:37.850282 15346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:37.853245 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.851037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:58677 (local address 127.10.109.126:46431)
0620 18:34:37.851112 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.851114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.851374 (+ 260us) server_negotiation.cc:407] Connection header received
0620 18:34:37.851891 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.851893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.851899 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.851969 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.853057 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.853058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.853058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:37.853062 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.853075 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:37.853078 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.853124 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.853184 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.853196 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:34:37.853535 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.850999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46431 (local address 127.10.109.65:58677)
0620 18:34:37.851203 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.851264 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.851788 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.852023 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.852026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.852033 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:37.852923 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:37.852923 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:37.852951 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:37.852978 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:37.853124 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:37.853126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:37.853128 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.853181 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.853226 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260620 18:34:37.854079 15474 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46431
I20260620 18:34:37.854156 15474 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:37.854230 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15346
I20260620 18:34:37.854295 15474 heartbeater.cc:511] Master 127.10.109.126:46431 requested a full tablet report, sending...
I20260620 18:34:37.854351 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:37.854571 10677 external_mini_cluster.cc:1428] Setting key 22f097d44ebc0ac028fef650ef3f5e43
I20260620 18:34:37.854842 15286 ts_manager.cc:194] Registered new tserver with Master: 932b3870f2e647918d2f71dc8e227d78 (127.10.109.65:37959)
I20260620 18:34:37.855616 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--builtin_ntp_servers=127.10.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:34:37.933419 15477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:37.933647 15477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:37.933682 15477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:37.934803 15477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:37.934867 15477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:37.934908 15477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:37.936224 15477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:37.936479 15477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:37.936754 15477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:37.941834 15485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.941969 15477 server_base.cc:1034] running on GCE node
W20260620 18:34:37.941926 15482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:37.941823 15483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:37.942230 15477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:37.942488 15477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:37.943677 15477 hybrid_clock.cc:648] HybridClock initialized: now 1781980477943681 us; error 47 us; skew 500 ppm
I20260620 18:34:37.944898 15477 webserver.cc:460] Webserver started at http://127.10.109.66:36523/ using document root <none> and password file <none>
I20260620 18:34:37.945127 15477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:37.945195 15477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:37.945374 15477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:37.946682 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "9966697d3f984aa09173fd4b1ab3a120"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bde070de614b6eb337a7c8be6b44362e"
server_key_iv: "2540b01a6aa373b9d632c0a14cc0cbe2"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.947156 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "9966697d3f984aa09173fd4b1ab3a120"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bde070de614b6eb337a7c8be6b44362e"
server_key_iv: "2540b01a6aa373b9d632c0a14cc0cbe2"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.948920 15477 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:34:37.949654 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.949932 15477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:37.950079 15477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "9966697d3f984aa09173fd4b1ab3a120"
format_stamp: "Formatted at 2026-06-20 18:34:37 on dist-test-slave-3jql"
server_key: "bde070de614b6eb337a7c8be6b44362e"
server_key_iv: "2540b01a6aa373b9d632c0a14cc0cbe2"
server_key_version: "encryptionkey@0"
I20260620 18:34:37.950181 15477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:37.954082 15477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:37.954344 15477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:37.954485 15477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:37.954694 15477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:37.955021 15477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:37.955075 15477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.955114 15477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:37.955142 15477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:37.960780 15477 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:40089
I20260620 18:34:37.960817 15604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:40089 every 8 connection(s)
I20260620 18:34:37.961172 15477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:37.964313 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.961899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:48187 (local address 127.10.109.126:46431)
0620 18:34:37.961980 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:37.961982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:37.962181 (+ 199us) server_negotiation.cc:407] Connection header received
0620 18:34:37.962617 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:37.962618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:37.962623 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:37.962674 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:37.963910 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:37.963912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:37.963912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:37.963916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:37.963931 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:37.963935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:37.964001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:37.964247 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:34:37.964261 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:34:37.964354 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:37.961879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46431 (local address 127.10.109.66:48187)
0620 18:34:37.962069 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:37.962100 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:37.962538 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:37.962726 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:37.962729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:37.962791 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:37.963763 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:37.963763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:37.963789 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:37.963815 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:37.964070 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:37.964076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:37.964079 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:37.964157 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:34:37.964205 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 18:34:37.965169 15605 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46431
I20260620 18:34:37.965286 15605 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:37.965435 15605 heartbeater.cc:511] Master 127.10.109.126:46431 requested a full tablet report, sending...
I20260620 18:34:37.965783 15285 ts_manager.cc:194] Registered new tserver with Master: 9966697d3f984aa09173fd4b1ab3a120 (127.10.109.66:40089)
I20260620 18:34:37.969903 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15477
I20260620 18:34:37.970002 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:37.970147 10677 external_mini_cluster.cc:1428] Setting key 97ca5af44b6144991d8de294416e1c04
I20260620 18:34:37.971107 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--builtin_ntp_servers=127.10.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:34:38.051746 15608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:38.051939 15608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:38.051960 15608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:38.053225 15608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:38.053274 15608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:38.053301 15608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:38.054629 15608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:38.054868 15608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:38.055099 15608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:38.059901 15614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.059988 15616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.060117 15608 server_base.cc:1034] running on GCE node
W20260620 18:34:38.059999 15613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.060406 15608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:38.060597 15608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:38.061739 15608 hybrid_clock.cc:648] HybridClock initialized: now 1781980478061725 us; error 27 us; skew 500 ppm
I20260620 18:34:38.063102 15608 webserver.cc:460] Webserver started at http://127.10.109.67:40903/ using document root <none> and password file <none>
I20260620 18:34:38.063351 15608 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:38.063400 15608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:38.063586 15608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:38.064594 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "4f9a192764ba42729ec90922871556fb"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6d0ebe94c13b2680aca1fcc1b57e1ff0"
server_key_iv: "5bba091b8843feb2669254fe5571b705"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.064944 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "4f9a192764ba42729ec90922871556fb"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6d0ebe94c13b2680aca1fcc1b57e1ff0"
server_key_iv: "5bba091b8843feb2669254fe5571b705"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.066448 15608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:38.067291 15622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.067494 15608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:38.067574 15608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "4f9a192764ba42729ec90922871556fb"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6d0ebe94c13b2680aca1fcc1b57e1ff0"
server_key_iv: "5bba091b8843feb2669254fe5571b705"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.067624 15608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:38.070694 15608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:38.070951 15608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:38.071044 15608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:38.071188 15608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:38.071410 15608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:38.071439 15608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.071458 15608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:38.071471 15608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.076743 15608 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:43239
I20260620 18:34:38.076841 15735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:43239 every 8 connection(s)
I20260620 18:34:38.077118 15608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:38.080005 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.077917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:54553 (local address 127.10.109.126:46431)
0620 18:34:38.078012 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.078014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.078108 (+ 94us) server_negotiation.cc:407] Connection header received
0620 18:34:38.078551 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.078552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.078558 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.078616 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.079763 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.079764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.079765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.079769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.079783 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.079787 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.079837 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.079914 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.079927 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:34:38.080139 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.077812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46431 (local address 127.10.109.67:54553)
0620 18:34:38.078001 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.078032 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.078445 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.078692 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.078695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.078703 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.079626 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.079627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.079652 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.079678 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.079846 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.079850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.079853 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.079910 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.079954 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260620 18:34:38.080641 15736 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46431
I20260620 18:34:38.080726 15736 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:38.080828 15736 heartbeater.cc:511] Master 127.10.109.126:46431 requested a full tablet report, sending...
I20260620 18:34:38.081171 15285 ts_manager.cc:194] Registered new tserver with Master: 4f9a192764ba42729ec90922871556fb (127.10.109.67:43239)
I20260620 18:34:38.085618 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15608
I20260620 18:34:38.085712 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:38.085844 10677 external_mini_cluster.cc:1428] Setting key 472494beeb110caa868bd6eb9f5435da
I20260620 18:34:38.086727 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:38.089205 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.088090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.10.109.126:46431)
0620 18:34:38.088162 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.088164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.088242 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:34:38.088327 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.088328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.088335 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.088387 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.088949 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.088950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.088950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.088954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.088967 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.089001 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.089046 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.089077 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.089090 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:34:38.090917 15285 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:38.092417 15285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:38.096748 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.095095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40089 (local address 127.0.0.1:33122)
0620 18:34:38.095529 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.095539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.095601 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.095789 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.095794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.095800 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.096250 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.096251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.096267 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.096292 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.096560 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.096566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.096569 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.096646 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.096662 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":88,"threads_started":1}
I20260620 18:34:38.097038 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.095351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.10.109.66:40089)
0620 18:34:38.095546 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.095550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.095604 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:34:38.095677 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.095680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.095692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.095769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.096395 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.096403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.096403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.096408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.096445 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.096468 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.096530 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.096660 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.096681 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260620 18:34:38.097141 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.094961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37959 (local address 127.0.0.1:39176)
0620 18:34:38.095310 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.095366 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.095431 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.095976 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.095977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.096208 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.096579 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.096580 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.096586 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.096597 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.096988 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.096989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.096991 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.097057 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.097072 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":73,"threads_started":1}
I20260620 18:34:38.097406 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.095111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.10.109.65:37959)
0620 18:34:38.095809 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.095813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.095824 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:38.095851 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.095853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.095863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.095956 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.096698 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.096705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.096706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.096710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.096743 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.096771 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.096994 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.097066 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.097083 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":94,"threads_started":1}
I20260620 18:34:38.097726 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.095209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:43239 (local address 127.0.0.1:36084)
0620 18:34:38.095597 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.095607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.095655 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.096430 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.096431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.096433 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.097048 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.097048 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.097055 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.097068 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.097596 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.097597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.097598 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.097637 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.097650 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":75,"threads_started":1}
I20260620 18:34:38.097764 15539 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 (DEFAULT_TABLE table=test-table [id=d4b29b233c6f4e799a85f98db271d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:38.097919 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.095293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.10.109.67:43239)
0620 18:34:38.095435 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.095439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.096285 (+ 846us) server_negotiation.cc:407] Connection header received
0620 18:34:38.096317 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.096320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.096330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.096420 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.097432 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.097438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.097439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.097443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.097489 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.097514 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.097573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.097824 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.097841 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260620 18:34:38.097975 15408 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 (DEFAULT_TABLE table=test-table [id=d4b29b233c6f4e799a85f98db271d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:38.098167 15539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2c2b5af0b4599ad9967ba4a3c8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.098305 15408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2c2b5af0b4599ad9967ba4a3c8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.099341 15670 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 (DEFAULT_TABLE table=test-table [id=d4b29b233c6f4e799a85f98db271d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:38.099684 15670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2c2b5af0b4599ad9967ba4a3c8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.101645 15751 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: Bootstrap starting.
I20260620 18:34:38.102381 15751 tablet_bootstrap.cc:654] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.102448 15752 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Bootstrap starting.
I20260620 18:34:38.102881 15751 log.cc:824] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.103153 15752 tablet_bootstrap.cc:654] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.103223 15750 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Bootstrap starting.
I20260620 18:34:38.103497 15752 log.cc:824] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.103689 15751 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: No bootstrap required, opened a new log
I20260620 18:34:38.103775 15751 ts_tablet_manager.cc:1397] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:38.103816 15750 tablet_bootstrap.cc:654] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.104115 15750 log.cc:824] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.104318 15752 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: No bootstrap required, opened a new log
I20260620 18:34:38.104396 15752 ts_tablet_manager.cc:1397] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:34:38.104825 15750 tablet_bootstrap.cc:492] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: No bootstrap required, opened a new log
I20260620 18:34:38.104907 15750 ts_tablet_manager.cc:1397] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:38.106890 15751 raft_consensus.cc:348] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.107074 15751 raft_consensus.cc:374] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.106990 15750 raft_consensus.cc:348] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.107112 15751 raft_consensus.cc:729] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932b3870f2e647918d2f71dc8e227d78, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.107141 15750 raft_consensus.cc:374] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.107189 15750 raft_consensus.cc:729] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9966697d3f984aa09173fd4b1ab3a120, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.107226 15751 consensus_queue.cc:260] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.107348 15750 consensus_queue.cc:260] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.107300 15752 raft_consensus.cc:348] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.107445 15752 raft_consensus.cc:374] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.107491 15751 ts_tablet_manager.cc:1428] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 18:34:38.107489 15752 raft_consensus.cc:729] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9a192764ba42729ec90922871556fb, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.107600 15750 ts_tablet_manager.cc:1428] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:38.107604 15474 heartbeater.cc:503] Master 127.10.109.126:46431 was elected leader, sending a full tablet report...
I20260620 18:34:38.107791 15605 heartbeater.cc:503] Master 127.10.109.126:46431 was elected leader, sending a full tablet report...
I20260620 18:34:38.108023 15752 consensus_queue.cc:260] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.108299 15752 ts_tablet_manager.cc:1428] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 18:34:38.108469 15736 heartbeater.cc:503] Master 127.10.109.126:46431 was elected leader, sending a full tablet report...
I20260620 18:34:38.177117 15758 raft_consensus.cc:482] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:38.177245 15758 raft_consensus.cc:504] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.177632 15758 leader_election.cc:288] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 932b3870f2e647918d2f71dc8e227d78 (127.10.109.65:37959), 9966697d3f984aa09173fd4b1ab3a120 (127.10.109.66:40089)
I20260620 18:34:38.179039 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.177814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40089 (local address 127.10.109.67:58269)
0620 18:34:38.177871 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.177883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.177921 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.178116 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.178117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.178120 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.178614 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.178614 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.178622 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.178635 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.178889 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.178890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.178892 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.178944 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.178958 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:34:38.179100 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.177937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:58269 (local address 127.10.109.66:40089)
0620 18:34:38.177993 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.177995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.178003 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:38.178035 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.178035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.178040 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.178108 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.178770 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.178773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.178773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.178779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.178799 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.178804 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.178873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.178998 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.179014 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:38.179164 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.177878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:54439 (local address 127.10.109.65:37959)
0620 18:34:38.177924 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.177926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.178080 (+ 154us) server_negotiation.cc:407] Connection header received
0620 18:34:38.178142 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.178143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.178150 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.178224 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.178922 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.178924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.178924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.178929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.178949 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.178954 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.179011 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.179071 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.179085 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:38.179351 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.177828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37959 (local address 127.10.109.67:54439)
0620 18:34:38.177997 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.178012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.178083 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.178227 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.178228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.178231 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:38.178810 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:34:38.178811 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:38.178818 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:34:38.178830 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:38.179018 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:38.179019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:38.179019 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 18:34:38.179071 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:34:38.179272 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260620 18:34:38.179370 15559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2c2b5af0b4599ad9967ba4a3c8938" candidate_uuid: "4f9a192764ba42729ec90922871556fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9966697d3f984aa09173fd4b1ab3a120" is_pre_election: true
I20260620 18:34:38.179515 15559 raft_consensus.cc:2454] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f9a192764ba42729ec90922871556fb in term 0.
I20260620 18:34:38.179740 15623 leader_election.cc:302] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f9a192764ba42729ec90922871556fb, 9966697d3f984aa09173fd4b1ab3a120; no voters:
I20260620 18:34:38.179726 15428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2c2b5af0b4599ad9967ba4a3c8938" candidate_uuid: "4f9a192764ba42729ec90922871556fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932b3870f2e647918d2f71dc8e227d78" is_pre_election: true
I20260620 18:34:38.179864 15428 raft_consensus.cc:2454] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f9a192764ba42729ec90922871556fb in term 0.
I20260620 18:34:38.179949 15758 raft_consensus.cc:2781] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:38.180034 15758 raft_consensus.cc:482] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:38.180068 15758 raft_consensus.cc:3037] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.180761 15758 raft_consensus.cc:504] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.180905 15758 leader_election.cc:288] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [CANDIDATE]: Term 1 election: Requested vote from peers 932b3870f2e647918d2f71dc8e227d78 (127.10.109.65:37959), 9966697d3f984aa09173fd4b1ab3a120 (127.10.109.66:40089)
I20260620 18:34:38.181066 15428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2c2b5af0b4599ad9967ba4a3c8938" candidate_uuid: "4f9a192764ba42729ec90922871556fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932b3870f2e647918d2f71dc8e227d78"
I20260620 18:34:38.181061 15559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2c2b5af0b4599ad9967ba4a3c8938" candidate_uuid: "4f9a192764ba42729ec90922871556fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9966697d3f984aa09173fd4b1ab3a120"
I20260620 18:34:38.181139 15428 raft_consensus.cc:3037] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.181141 15559 raft_consensus.cc:3037] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.182006 15559 raft_consensus.cc:2454] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9a192764ba42729ec90922871556fb in term 1.
I20260620 18:34:38.182005 15428 raft_consensus.cc:2454] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f9a192764ba42729ec90922871556fb in term 1.
I20260620 18:34:38.182185 15625 leader_election.cc:302] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f9a192764ba42729ec90922871556fb, 932b3870f2e647918d2f71dc8e227d78; no voters:
I20260620 18:34:38.182322 15758 raft_consensus.cc:2781] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:38.182502 15758 raft_consensus.cc:686] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 1 LEADER]: Becoming Leader. State: Replica: 4f9a192764ba42729ec90922871556fb, State: Running, Role: LEADER
I20260620 18:34:38.182602 15758 consensus_queue.cc:237] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } }
I20260620 18:34:38.183405 15286 catalog_manager.cc:5495] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f9a192764ba42729ec90922871556fb (127.10.109.67). New cstate: current_term: 1 leader_uuid: "4f9a192764ba42729ec90922871556fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f9a192764ba42729ec90922871556fb" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:38.189056 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.187990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.10.109.67:43239)
0620 18:34:38.188058 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.188060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.188069 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:38.188138 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.188139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.188148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.188197 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.188744 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.188745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.188745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.188749 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.188761 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.188765 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.188817 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.188951 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.188980 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:34:38.191174 15559 raft_consensus.cc:1261] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 1 FOLLOWER]: Refusing update from remote peer 4f9a192764ba42729ec90922871556fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:38.191174 15428 raft_consensus.cc:1261] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78 [term 1 FOLLOWER]: Refusing update from remote peer 4f9a192764ba42729ec90922871556fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:38.191506 15760 consensus_queue.cc:1035] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "932b3870f2e647918d2f71dc8e227d78" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:38.191607 15758 consensus_queue.cc:1035] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9966697d3f984aa09173fd4b1ab3a120" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:38.196075 15286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:38.196162 15286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:38.196890 15286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c78be1e0293a46d8bf582d4aaaecb129: Sending DeleteTablet for 3 replicas of tablet 20f2c2b5af0b4599ad9967ba4a3c8938
I20260620 18:34:38.197204 15670 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:38 UTC) from {username='slave'} at 127.0.0.1:36084
I20260620 18:34:38.197240 15408 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:38 UTC) from {username='slave'} at 127.0.0.1:39176
I20260620 18:34:38.197755 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15346
I20260620 18:34:38.197731 15539 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2c2b5af0b4599ad9967ba4a3c8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:38 UTC) from {username='slave'} at 127.0.0.1:33122
I20260620 18:34:38.198096 15773 tablet_replica.cc:330] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 932b3870f2e647918d2f71dc8e227d78: stopping tablet replica
I20260620 18:34:38.198344 15774 tablet_replica.cc:330] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: stopping tablet replica
I20260620 18:34:38.198524 15774 raft_consensus.cc:2229] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:38.198854 15774 raft_consensus.cc:2258] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:38.199532 15774 ts_tablet_manager.cc:1905] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:38.201309 15774 ts_tablet_manager.cc:1918] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:38.201431 15774 log.cc:1196] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/20f2c2b5af0b4599ad9967ba4a3c8938
I20260620 18:34:38.201658 15772 tablet_replica.cc:330] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: stopping tablet replica
I20260620 18:34:38.201749 15772 raft_consensus.cc:2229] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:38.201865 15772 raft_consensus.cc:2258] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:38.202016 15774 ts_tablet_manager.cc:1939] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 9966697d3f984aa09173fd4b1ab3a120: Deleting consensus metadata
I20260620 18:34:38.202291 15772 ts_tablet_manager.cc:1905] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:38.202644 15270 catalog_manager.cc:4841] TS 9966697d3f984aa09173fd4b1ab3a120 (127.10.109.66:40089): tablet 20f2c2b5af0b4599ad9967ba4a3c8938 (table test-table [id=d4b29b233c6f4e799a85f98db271d75e]) successfully deleted
I20260620 18:34:38.203948 15772 ts_tablet_manager.cc:1918] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:38.204002 15772 log.cc:1196] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/20f2c2b5af0b4599ad9967ba4a3c8938
I20260620 18:34:38.204234 15772 ts_tablet_manager.cc:1939] T 20f2c2b5af0b4599ad9967ba4a3c8938 P 4f9a192764ba42729ec90922871556fb: Deleting consensus metadata
I20260620 18:34:38.204584 15272 catalog_manager.cc:4841] TS 4f9a192764ba42729ec90922871556fb (127.10.109.67:43239): tablet 20f2c2b5af0b4599ad9967ba4a3c8938 (table test-table [id=d4b29b233c6f4e799a85f98db271d75e]) successfully deleted
W20260620 18:34:38.210647 15625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:37959 (error 108)
I20260620 18:34:38.210937 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15477
W20260620 18:34:38.214403 15272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:37959 (error 108)
I20260620 18:34:38.214949 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.214830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37959 (local address 127.0.0.1:39190)
0620 18:34:38.214876 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.214915 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:37959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 18:34:38.214998 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.214705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:37959 (local address 127.10.109.67:33573)
0620 18:34:38.214808 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.214966 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:37959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:34:38.215065 15272 catalog_manager.cc:4584] TS 932b3870f2e647918d2f71dc8e227d78 (127.10.109.65:37959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20f2c2b5af0b4599ad9967ba4a3c8938: Network error: Client connection negotiation failed: client connection to 127.10.109.65:37959: connect: Connection refused (error 111)
I20260620 18:34:38.219475 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15608
I20260620 18:34:38.224987 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15255
2026-06-20T18:34:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (663 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:34:38.247210 10677 mini_oidc.cc:152] Starting JWKS server
I20260620 18:34:38.248891 10677 webserver.cc:460] Webserver started at https://127.0.0.1:38047/ using document root <none> and password file <none>
I20260620 18:34:38.249018 10677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:34:38.249202 10677 webserver.cc:460] Webserver started at http://127.0.0.1:33931/ using document root <none> and password file <none>
2026-06-20T18:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:38Z Disabled control of system clock
I20260620 18:34:38.257714 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41583
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:41583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38047/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/testchainca.cert with env {}
W20260620 18:34:38.339128 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:38.339325 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:38.339346 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:38.340436 15782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:38.340492 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:38.340505 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:38.340518 15782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:38.340529 15782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:34:38.340548 15782 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:34:38.340559 15782 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38047/jwks.json
I20260620 18:34:38.342108 15782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:41583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38047/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:38.342298 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:38.342447 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:38.347249 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.347250 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.347265 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.347296 15782 server_base.cc:1034] running on GCE node
I20260620 18:34:38.347720 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:38.348016 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:38.349192 15782 hybrid_clock.cc:648] HybridClock initialized: now 1781980478349180 us; error 40 us; skew 500 ppm
I20260620 18:34:38.350690 15782 webserver.cc:460] Webserver started at http://127.10.109.126:39475/ using document root <none> and password file <none>
I20260620 18:34:38.350941 15782 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:38.350992 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:38.351079 15782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:38.352134 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "c8e35966198649949cf5dbef0463af14"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6e59684e50d60945fa68a49efeb2ee13"
server_key_iv: "a9541f476b9c82c9b0425cc6982b9f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.352435 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "c8e35966198649949cf5dbef0463af14"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6e59684e50d60945fa68a49efeb2ee13"
server_key_iv: "a9541f476b9c82c9b0425cc6982b9f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.353636 15782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:38.354274 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.354430 15782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:38.354516 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "c8e35966198649949cf5dbef0463af14"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "6e59684e50d60945fa68a49efeb2ee13"
server_key_iv: "a9541f476b9c82c9b0425cc6982b9f1d"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.354568 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:38.368664 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:38.368916 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:38.369014 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:38.372795 15782 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:41583
I20260620 18:34:38.372891 15848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:41583 every 8 connection(s)
I20260620 18:34:38.373194 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:38.374161 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.376526 15849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: Bootstrap starting.
I20260620 18:34:38.377144 15849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.377441 15849 log.cc:824] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.378144 15849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: No bootstrap required, opened a new log
I20260620 18:34:38.379789 15849 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } }
I20260620 18:34:38.379925 15849 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.379946 15849 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e35966198649949cf5dbef0463af14, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.380098 15849 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } }
I20260620 18:34:38.380174 15849 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:38.380216 15849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:38.380267 15849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.380797 15849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } }
I20260620 18:34:38.380909 15849 leader_election.cc:302] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e35966198649949cf5dbef0463af14; no voters:
I20260620 18:34:38.381102 15849 leader_election.cc:288] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:38.381168 15852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:38.381357 15852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [term 1 LEADER]: Becoming Leader. State: Replica: c8e35966198649949cf5dbef0463af14, State: Running, Role: LEADER
I20260620 18:34:38.381356 15849 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:38.381527 15852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } }
I20260620 18:34:38.381804 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15782
I20260620 18:34:38.381925 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:38.381996 15853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e35966198649949cf5dbef0463af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } } }
I20260620 18:34:38.382077 15854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e35966198649949cf5dbef0463af14. Latest consensus state: current_term: 1 leader_uuid: "c8e35966198649949cf5dbef0463af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e35966198649949cf5dbef0463af14" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41583 } } }
I20260620 18:34:38.382136 15854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.382138 10677 external_mini_cluster.cc:1428] Setting key 447342647afc236fd0428eb4d498c439
I20260620 18:34:38.382084 15853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14 [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:38.383934 15870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:38.384007 15870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:38.384073 15856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:38.384238 15856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:38.385766 15856 catalog_manager.cc:1347] Generated new cluster ID: df2f42808b9e4f1ca8b8094fba6e3f04
I20260620 18:34:38.385821 15856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:38.388397 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.382590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.10.109.126:41583)
0620 18:34:38.382855 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.382864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.382899 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:34:38.383021 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.383027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.383076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.383209 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.383473 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.384908 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.386204 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.386845 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.386961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.387709 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.387713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.387713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.387915 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.387968 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.387980 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.388040 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.388141 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.388209 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260620 18:34:38.389282 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:38.400480 15856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:38.401180 15856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:38.407893 15856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e35966198649949cf5dbef0463af14: Generated new TSK 0
I20260620 18:34:38.408072 15856 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:34:38.437676 15879 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:34:38.437783 15879 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 18:34:38.438360 15878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:38.421343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41583 (local address 127.0.0.1:33312)
0620 18:34:38.421520 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.421530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.421589 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.421812 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.421813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.421815 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:34:38.421898 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.421900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.422850 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.422851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.423434 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.423436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.423914 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.423915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.423962 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.423978 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:34:38.437944 (+ 13966us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:34:38.438049 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.109.126:41583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":100,"threads_started":1}
I20260620 18:34:38.438783 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.421421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.10.109.126:41583)
0620 18:34:38.421472 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.421474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.421594 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:34:38.421668 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.421670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.421683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.421763 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:34:38.421974 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.422733 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.423516 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.423813 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.423883 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.424087 (+ 204us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:34:38.437806 (+ 13719us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:34:38.438584 (+ 778us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 18:34:38.438839 15860 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.109.126:41583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:34:38.438984 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15782
2026-06-20T18:34:38Z chronyd exiting
I20260620 18:34:38.474587 10677 test_util.cc:175] -----------------------------------------------
I20260620 18:34:38.474684 10677 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980452801806-10677-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:34:38.475528 10677 mini_oidc.cc:152] Starting JWKS server
I20260620 18:34:38.476212 10677 webserver.cc:460] Webserver started at https://127.0.0.1:35921/ using document root <none> and password file <none>
I20260620 18:34:38.476277 10677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:34:38.476466 10677 webserver.cc:460] Webserver started at http://127.0.0.1:33505/ using document root <none> and password file <none>
2026-06-20T18:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:38Z Disabled control of system clock
I20260620 18:34:38.483769 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33307
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:33307
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/testchainca.cert with env {}
W20260620 18:34:38.561265 15885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:38.561501 15885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:38.561537 15885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:38.562667 15885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:38.562733 15885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:38.562775 15885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:38.562798 15885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:38.562826 15885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:34:38.562847 15885 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:34:38.562868 15885 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35921/jwks.json
I20260620 18:34:38.564329 15885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:33307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:38.564600 15885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:38.564805 15885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:38.569473 15891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.569479 15890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.569479 15893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.569669 15885 server_base.cc:1034] running on GCE node
I20260620 18:34:38.569936 15885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:38.570225 15885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:38.571385 15885 hybrid_clock.cc:648] HybridClock initialized: now 1781980478571354 us; error 49 us; skew 500 ppm
I20260620 18:34:38.572634 15885 webserver.cc:460] Webserver started at http://127.10.109.126:43743/ using document root <none> and password file <none>
I20260620 18:34:38.572873 15885 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:38.572937 15885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:38.573079 15885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:38.574070 15885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "dd3fe73118db4bd898c51620ff0e33bc"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "412494e3da67fe0a1b217b23183fbe98"
server_key_iv: "acc643fc1613a09d59e1935cf3a48f17"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.574470 15885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "dd3fe73118db4bd898c51620ff0e33bc"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "412494e3da67fe0a1b217b23183fbe98"
server_key_iv: "acc643fc1613a09d59e1935cf3a48f17"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.575796 15885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:38.576535 15899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.576706 15885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:38.576797 15885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "dd3fe73118db4bd898c51620ff0e33bc"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "412494e3da67fe0a1b217b23183fbe98"
server_key_iv: "acc643fc1613a09d59e1935cf3a48f17"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.576879 15885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:38.599373 15885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:38.599670 15885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:38.599815 15885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:38.603710 15885 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:33307
I20260620 18:34:38.603760 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:33307 every 8 connection(s)
I20260620 18:34:38.604100 15885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:38.604723 15952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.607436 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc: Bootstrap starting.
I20260620 18:34:38.607753 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15885
I20260620 18:34:38.607878 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:38.608048 10677 external_mini_cluster.cc:1428] Setting key 6b0ebec9f04dd420310b5109321594b2
I20260620 18:34:38.608103 15952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.608460 15952 log.cc:824] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.609306 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc: No bootstrap required, opened a new log
I20260620 18:34:38.611191 15952 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } }
I20260620 18:34:38.611339 15952 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.611418 15952 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3fe73118db4bd898c51620ff0e33bc, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.611574 15952 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } }
I20260620 18:34:38.611672 15952 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:38.611716 15952 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:38.611770 15952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.612313 15952 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } }
I20260620 18:34:38.612429 15952 leader_election.cc:302] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd3fe73118db4bd898c51620ff0e33bc; no voters:
I20260620 18:34:38.612598 15952 leader_election.cc:288] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:38.612764 15958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:38.612872 15952 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:38.612927 15958 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [term 1 LEADER]: Becoming Leader. State: Replica: dd3fe73118db4bd898c51620ff0e33bc, State: Running, Role: LEADER
I20260620 18:34:38.612999 15958 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } }
I20260620 18:34:38.613605 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd3fe73118db4bd898c51620ff0e33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } } }
I20260620 18:34:38.613672 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.608551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.10.109.126:33307)
0620 18:34:38.608747 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.608758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.608782 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:34:38.609246 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.609252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.609305 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.609462 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.609661 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.610518 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.611420 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.611971 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.612110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.612775 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.612779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.612780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.613010 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.613047 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.613060 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.613369 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.613401 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.613476 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260620 18:34:38.613690 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.613833 15959 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd3fe73118db4bd898c51620ff0e33bc. Latest consensus state: current_term: 1 leader_uuid: "dd3fe73118db4bd898c51620ff0e33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3fe73118db4bd898c51620ff0e33bc" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33307 } } }
I20260620 18:34:38.613888 15959 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.614511 15966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:38.614573 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:38.615105 15966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:38.616483 15966 catalog_manager.cc:1347] Generated new cluster ID: 4bfb83a2b1ca48f3992150028959d874
I20260620 18:34:38.616537 15966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:38.624643 15966 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:38.625303 15966 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:38.635646 15966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd3fe73118db4bd898c51620ff0e33bc: Generated new TSK 0
I20260620 18:34:38.635818 15966 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:34:38.658432 15982 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:34:38.658615 15982 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 18:34:38.658949 15981 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:38.644300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:33307 (local address 127.0.0.1:41010)
0620 18:34:38.644473 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.644483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.644536 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.644682 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.644683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.644686 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:34:38.644729 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.644730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.645659 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.645659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.646235 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.646237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.646716 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.646716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.646807 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.646812 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:34:38.658732 (+ 11920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:34:38.658844 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.109.126:33307: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260620 18:34:38.659216 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.644375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.10.109.126:33307)
0620 18:34:38.644434 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.644436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.644537 (+ 101us) server_negotiation.cc:407] Connection header received
0620 18:34:38.644587 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.644588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.644601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.644678 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:34:38.644787 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.645573 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.646314 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.646639 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.646710 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.646901 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:34:38.658570 (+ 11669us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:34:38.659152 (+ 582us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41010: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:34:38.659292 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15885
2026-06-20T18:34:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:34:38.696319 10677 mini_oidc.cc:152] Starting JWKS server
I20260620 18:34:38.696965 10677 webserver.cc:460] Webserver started at https://127.0.0.1:34781/ using document root <none> and password file <none>
I20260620 18:34:38.697006 10677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:34:38.697139 10677 webserver.cc:460] Webserver started at http://127.0.0.1:34053/ using document root <none> and password file <none>
2026-06-20T18:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:38Z Disabled control of system clock
I20260620 18:34:38.704221 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34921
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:34921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34781/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:34:38.782238 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:38.782462 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:38.782500 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:38.783627 15988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:38.783692 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:38.783715 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:38.783743 15988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:38.783771 15988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:34:38.783793 15988 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:34:38.783823 15988 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34781/jwks.json
I20260620 18:34:38.785123 15988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:34921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34781/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:38.785377 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:38.785640 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:38.790408 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.790407 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:38.790477 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:38.790699 15988 server_base.cc:1034] running on GCE node
I20260620 18:34:38.790939 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:38.791218 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:38.792356 15988 hybrid_clock.cc:648] HybridClock initialized: now 1781980478792334 us; error 36 us; skew 500 ppm
I20260620 18:34:38.793614 15988 webserver.cc:460] Webserver started at http://127.10.109.126:38679/ using document root <none> and password file <none>
I20260620 18:34:38.793835 15988 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:38.793881 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:38.794047 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:38.795106 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "5e7f6197b4cb424882b6df9af628c6fe"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "3d36af424852648f3f81db8ca7f27d6c"
server_key_iv: "33dd2285be6958c6815c8a805ececa5b"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.795439 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "5e7f6197b4cb424882b6df9af628c6fe"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "3d36af424852648f3f81db8ca7f27d6c"
server_key_iv: "33dd2285be6958c6815c8a805ececa5b"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.796660 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:38.797346 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:38.797523 15988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:38.797616 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "5e7f6197b4cb424882b6df9af628c6fe"
format_stamp: "Formatted at 2026-06-20 18:34:38 on dist-test-slave-3jql"
server_key: "3d36af424852648f3f81db8ca7f27d6c"
server_key_iv: "33dd2285be6958c6815c8a805ececa5b"
server_key_version: "encryptionkey@0"
I20260620 18:34:38.797704 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:38.820590 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:38.820878 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:38.821033 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:38.825001 15988 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:34921
I20260620 18:34:38.825033 16054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:34921 every 8 connection(s)
I20260620 18:34:38.825349 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:38.826006 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:38.828384 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 15988
I20260620 18:34:38.828519 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:38.828696 10677 external_mini_cluster.cc:1428] Setting key 171c856862784ea515abf1a68dd85746
I20260620 18:34:38.829253 16055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe: Bootstrap starting.
I20260620 18:34:38.829856 16055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:38.830199 16055 log.cc:824] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:38.830974 16055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe: No bootstrap required, opened a new log
I20260620 18:34:38.833098 16055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } }
I20260620 18:34:38.833272 16055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:38.833309 16055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6197b4cb424882b6df9af628c6fe, State: Initialized, Role: FOLLOWER
I20260620 18:34:38.833482 16055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } }
I20260620 18:34:38.833585 16055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:38.833626 16055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:38.833674 16055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:38.834313 16055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } }
I20260620 18:34:38.834453 16055 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7f6197b4cb424882b6df9af628c6fe; no voters:
I20260620 18:34:38.834610 16055 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:38.834795 16061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:38.834864 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.829244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.10.109.126:34921)
0620 18:34:38.829464 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.829471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.829495 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:34:38.829899 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.829904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.829999 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.830124 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:38.830402 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.831757 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.832507 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.833166 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.833300 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.834033 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:38.834038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:38.834038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:38.834245 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:38.834289 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:38.834306 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:38.834385 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:38.834459 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:34:38.834539 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260620 18:34:38.834893 16055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:38.834920 16061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [term 1 LEADER]: Becoming Leader. State: Replica: 5e7f6197b4cb424882b6df9af628c6fe, State: Running, Role: LEADER
I20260620 18:34:38.835681 16061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } }
I20260620 18:34:38.836048 16061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7f6197b4cb424882b6df9af628c6fe. Latest consensus state: current_term: 1 leader_uuid: "5e7f6197b4cb424882b6df9af628c6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } } }
I20260620 18:34:38.836170 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:38.836191 16061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.836406 16075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:38.836882 16075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:38.837111 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7f6197b4cb424882b6df9af628c6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f6197b4cb424882b6df9af628c6fe" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34921 } } }
I20260620 18:34:38.837244 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:38.838410 16075 catalog_manager.cc:1347] Generated new cluster ID: 233ad1ffc5fd4734b22157aa08547069
I20260620 18:34:38.838464 16075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:38.863168 16075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:38.863691 16075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:38.870639 16075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7f6197b4cb424882b6df9af628c6fe: Generated new TSK 0
I20260620 18:34:38.870838 16075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:38.884949 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:38.881201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.10.109.126:34921)
0620 18:34:38.881243 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:38.881245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:38.881332 (+ 87us) server_negotiation.cc:407] Connection header received
0620 18:34:38.881404 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:38.881405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:38.881414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:38.881472 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:34:38.881618 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.882107 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.883309 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.883501 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.883558 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.883783 (+ 225us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:34:38.884752 (+ 969us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:34:38.884885 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 18:34:38.885002 16058 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:34:38.885063 16084 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:38.881096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:34921 (local address 127.0.0.1:58320)
0620 18:34:38.881257 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:38.881268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:38.881338 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:38.881495 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:38.881496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:38.881499 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:34:38.881555 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.881557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.882206 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.882207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.883218 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:38.883222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:38.883590 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:38.883592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:38.883651 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:38.883656 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:34:38.884873 (+ 1217us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:34:38.884967 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.109.126:34921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260620 18:34:38.885448 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 15988
E20260620 18:34:38.897163 16085 webserver.cc:533] Webserver: sslize failed:
E20260620 18:34:38.897248 16085 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T18:34:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:38Z Disabled control of system clock
I20260620 18:34:38.932973 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35583
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:35583
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/testchainca.cert with env {}
W20260620 18:34:39.014957 16089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:39.015156 16089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:39.015177 16089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:39.016361 16089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:39.016407 16089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:39.016422 16089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:39.016433 16089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:39.016446 16089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:34:39.016458 16089 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:34:39.016469 16089 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:34:39.017998 16089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:35583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:39.018250 16089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:39.018472 16089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:34:39.023564 16089 server_base.cc:1034] running on GCE node
W20260620 18:34:39.023542 16094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:39.023665 16095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:39.023548 16097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:39.023972 16089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:39.024202 16089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:39.025342 16089 hybrid_clock.cc:648] HybridClock initialized: now 1781980479025327 us; error 34 us; skew 500 ppm
I20260620 18:34:39.026576 16089 webserver.cc:460] Webserver started at http://127.10.109.126:34481/ using document root <none> and password file <none>
I20260620 18:34:39.026862 16089 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:39.026918 16089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:39.027070 16089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:39.028059 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c"
format_stamp: "Formatted at 2026-06-20 18:34:39 on dist-test-slave-3jql"
server_key: "f26c23788f41343cc5d7cfb4ca8f3af3"
server_key_iv: "7d7c22d1bf655659f39cd838dca6ee7d"
server_key_version: "encryptionkey@0"
I20260620 18:34:39.028409 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c"
format_stamp: "Formatted at 2026-06-20 18:34:39 on dist-test-slave-3jql"
server_key: "f26c23788f41343cc5d7cfb4ca8f3af3"
server_key_iv: "7d7c22d1bf655659f39cd838dca6ee7d"
server_key_version: "encryptionkey@0"
I20260620 18:34:39.029708 16089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:39.030385 16103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:39.030575 16089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:39.030673 16089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c"
format_stamp: "Formatted at 2026-06-20 18:34:39 on dist-test-slave-3jql"
server_key: "f26c23788f41343cc5d7cfb4ca8f3af3"
server_key_iv: "7d7c22d1bf655659f39cd838dca6ee7d"
server_key_version: "encryptionkey@0"
I20260620 18:34:39.030781 16089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:39.050493 16089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:39.050877 16089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:39.051049 16089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:39.054975 16089 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:35583
I20260620 18:34:39.055011 16155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:35583 every 8 connection(s)
I20260620 18:34:39.055344 16089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:39.056270 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:39.057305 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16089
I20260620 18:34:39.057420 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:39.057598 10677 external_mini_cluster.cc:1428] Setting key d8460952a56b1e16effde59ee0a510d9
I20260620 18:34:39.059136 16156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c: Bootstrap starting.
I20260620 18:34:39.059772 16156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:39.060057 16156 log.cc:824] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:39.060756 16156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c: No bootstrap required, opened a new log
I20260620 18:34:39.063292 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:39.058118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.10.109.126:35583)
0620 18:34:39.058309 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:39.058323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:39.058348 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:39.058996 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:39.059003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:39.059063 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:39.059178 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:39.059400 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.060358 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.061151 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.061746 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.061871 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:39.062530 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:39.062535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:39.062536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:34:39.062788 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:39.062838 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:34:39.062855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:39.062951 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:39.063012 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:34:39.063090 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 18:34:39.063272 16156 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } }
I20260620 18:34:39.063441 16156 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:39.063506 16156 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3b3ea598ab4eeab97f8a21dee2ae2c, State: Initialized, Role: FOLLOWER
I20260620 18:34:39.063635 16156 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } }
I20260620 18:34:39.063730 16156 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:39.063776 16156 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:39.063827 16156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:39.064332 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:34:39.064529 16156 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } }
I20260620 18:34:39.064638 16156 leader_election.cc:302] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3b3ea598ab4eeab97f8a21dee2ae2c; no voters:
I20260620 18:34:39.064790 16156 leader_election.cc:288] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:39.064826 16161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:39.065006 16161 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [term 1 LEADER]: Becoming Leader. State: Replica: cd3b3ea598ab4eeab97f8a21dee2ae2c, State: Running, Role: LEADER
I20260620 18:34:39.065061 16156 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:39.065160 16161 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } }
I20260620 18:34:39.065582 16162 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3b3ea598ab4eeab97f8a21dee2ae2c. Latest consensus state: current_term: 1 leader_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } } }
I20260620 18:34:39.065578 16163 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3b3ea598ab4eeab97f8a21dee2ae2c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35583 } } }
I20260620 18:34:39.065829 16163 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:39.065742 16162 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:39.066226 16170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:39.066728 16170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:39.068282 16170 catalog_manager.cc:1347] Generated new cluster ID: d503d8b5084f4b4fb9f4199c925e1861
I20260620 18:34:39.068336 16170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:39.083009 16170 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:39.083508 16170 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:39.088233 16170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3b3ea598ab4eeab97f8a21dee2ae2c: Generated new TSK 0
I20260620 18:34:39.088402 16170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:39.182034 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:39.094626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.10.109.126:35583)
0620 18:34:39.094682 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:39.094684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:39.094787 (+ 103us) server_negotiation.cc:407] Connection header received
0620 18:34:39.094893 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:39.094894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:39.094908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:39.094983 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:34:39.095080 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.095879 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.096603 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.096922 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.096990 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:39.097119 (+ 129us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:34:39.181759 (+ 84640us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:34:39.181950 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 18:34:39.182109 16159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:34:39.182178 16185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:39.094558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35583 (local address 127.0.0.1:59486)
0620 18:34:39.094697 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:39.094707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:39.094815 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:39.094987 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:39.094988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:39.094990 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:34:39.095037 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:39.095039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.095948 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.095949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:39.096528 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:39.096529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:39.097000 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:39.097000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:39.097039 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:39.097043 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:34:39.181910 (+ 84867us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:34:39.182054 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.109.126:35583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 18:34:39.182500 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16089
2026-06-20T18:34:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (275 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:39 dist-test-slave-3jql krb5kdc[16197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:39 dist-test-slave-3jql krb5kdc[16197](info): set up 2 sockets
Jun 20 18:34:39 dist-test-slave-3jql krb5kdc[16197](info): commencing operation
krb5kdc: starting...
W20260620 18:34:40.459256 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:40 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:40.571421 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:39525
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:40.649083 16210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:40.649296 16210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:40.649330 16210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:40.650467 16210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:40.650530 16210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:40.650559 16210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:40.650586 16210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:40.650640 16210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:40.652122 16210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:40.652383 16210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:40.652601 16210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:34:40.657693 16210 server_base.cc:1034] running on GCE node
W20260620 18:34:40.657665 16216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.657640 16218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.657632 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:40.658113 16210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:40.658344 16210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:40.659490 16210 hybrid_clock.cc:648] HybridClock initialized: now 1781980480659466 us; error 39 us; skew 500 ppm
Jun 20 18:34:40 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:40.666529 16210 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:40.667065 16210 webserver.cc:460] Webserver started at http://127.10.109.126:44235/ using document root <none> and password file <none>
I20260620 18:34:40.667315 16210 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:40.667383 16210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:40.667536 16210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:40.668610 16210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "4473b87ba3134ebd9d9c41ddc76f887f"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "fd2474b7a6b3542efdc675ff1eb25e4e"
server_key_iv: "70a5f62fe5aed1cdb25e2b922a0b5089"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.668965 16210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "4473b87ba3134ebd9d9c41ddc76f887f"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "fd2474b7a6b3542efdc675ff1eb25e4e"
server_key_iv: "70a5f62fe5aed1cdb25e2b922a0b5089"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.670282 16210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:40.671105 16225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:40.671315 16210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:40.671417 16210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "4473b87ba3134ebd9d9c41ddc76f887f"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "fd2474b7a6b3542efdc675ff1eb25e4e"
server_key_iv: "70a5f62fe5aed1cdb25e2b922a0b5089"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.671501 16210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:40.713088 16210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:40.715518 16210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:40.715689 16210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:40.719661 16210 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:39525
I20260620 18:34:40.719728 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:39525 every 8 connection(s)
I20260620 18:34:40.720078 16210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:40.721303 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:40.723724 16278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f: Bootstrap starting.
I20260620 18:34:40.724309 16278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:40.724612 16278 log.cc:824] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:40.725334 16278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f: No bootstrap required, opened a new log
I20260620 18:34:40.725945 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16210
I20260620 18:34:40.726063 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:40.726286 10677 external_mini_cluster.cc:1428] Setting key d70e5e9d8c997e04d7ec5fd534987464
I20260620 18:34:40.727659 16278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } }
I20260620 18:34:40.727819 16278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:40.727927 16278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4473b87ba3134ebd9d9c41ddc76f887f, State: Initialized, Role: FOLLOWER
I20260620 18:34:40.728083 16278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } }
I20260620 18:34:40.728181 16278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:40.728223 16278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:40.728271 16278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:40.729087 16278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } }
I20260620 18:34:40.729204 16278 leader_election.cc:302] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4473b87ba3134ebd9d9c41ddc76f887f; no voters:
I20260620 18:34:40.729372 16278 leader_election.cc:288] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:40.729416 16283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:40.729544 16283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [term 1 LEADER]: Becoming Leader. State: Replica: 4473b87ba3134ebd9d9c41ddc76f887f, State: Running, Role: LEADER
I20260620 18:34:40.729636 16278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:40.729672 16283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } }
I20260620 18:34:40.730051 16283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4473b87ba3134ebd9d9c41ddc76f887f. Latest consensus state: current_term: 1 leader_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } } }
I20260620 18:34:40.730180 16283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:40.730399 16285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b87ba3134ebd9d9c41ddc76f887f" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 39525 } } }
I20260620 18:34:40.730485 16285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:40.730535 16290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:40.731086 16290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:40.733342 16300 mvcc.cc:205] Tried to move back new op lower bound from 7298992049078546432 to 7298992049069252608. Current Snapshot: MvccSnapshot[applied={T|T < 7298992049078546432}]
I20260620 18:34:40.733743 16290 catalog_manager.cc:1347] Generated new cluster ID: 7cf4379bc28040219bc11d596e96cf3f
I20260620 18:34:40.733795 16290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:40 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:40.737353 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:40.726844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.10.109.126:39525)
0620 18:34:40.727043 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:40.727052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:40.727084 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:34:40.727645 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:40.727651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:40.727695 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:40.727834 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:40.728605 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.729707 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.730796 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.731535 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.732031 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:40.734310 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:40.734314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:40.734314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:40.734324 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:40.735941 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:40.736307 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:40.736310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:40.736310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:40.736407 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:40.736614 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:40.736615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:40.736615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:40.736836 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:40.737008 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:40.737081 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:34:40.737210 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260620 18:34:40.739609 16290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:40.740082 16290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:40.749645 16290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4473b87ba3134ebd9d9c41ddc76f887f: Generated new TSK 0
I20260620 18:34:40.749809 16290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:40.779706 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:40.860775 16304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:40.861150 16304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:40.861200 16304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:40.862360 16304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:40.862398 16304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:40.862425 16304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:40.864027 16304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:40.864264 16304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:40.864485 16304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:40.869457 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.869422 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:40.869479 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:40.870101 16304 server_base.cc:1034] running on GCE node
I20260620 18:34:40.870277 16304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:40.870492 16304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:40.871655 16304 hybrid_clock.cc:648] HybridClock initialized: now 1781980480871631 us; error 39 us; skew 500 ppm
Jun 20 18:34:40 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:40.874362 16304 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:40.874869 16304 webserver.cc:460] Webserver started at http://127.10.109.65:33707/ using document root <none> and password file <none>
I20260620 18:34:40.875208 16304 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:40.875277 16304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:40.875416 16304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:40.876401 16304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "03d7727594e1459c946c3444311c934a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "10f775b1df9956fac4edf0d32562acf6"
server_key_iv: "b718f61597853fde3262b2e5d3b93729"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.876754 16304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "03d7727594e1459c946c3444311c934a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "10f775b1df9956fac4edf0d32562acf6"
server_key_iv: "b718f61597853fde3262b2e5d3b93729"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.878067 16304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:40.878890 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:40.879056 16304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:40.879155 16304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "03d7727594e1459c946c3444311c934a"
format_stamp: "Formatted at 2026-06-20 18:34:40 on dist-test-slave-3jql"
server_key: "10f775b1df9956fac4edf0d32562acf6"
server_key_iv: "b718f61597853fde3262b2e5d3b93729"
server_key_version: "encryptionkey@0"
I20260620 18:34:40.879240 16304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:40.898151 16304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:40.900084 16304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:40.900241 16304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:40.900471 16304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:40.900770 16304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:40.900823 16304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:40.900873 16304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:40.900903 16304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:40.906677 16304 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:38675
I20260620 18:34:40.906726 16432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:38675 every 8 connection(s)
I20260620 18:34:40.907038 16304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:40 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:40.914243 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16304
I20260620 18:34:40.914387 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:40.914587 10677 external_mini_cluster.cc:1428] Setting key 3add5f9bf5b37cd0eec7daf90f4886dc
I20260620 18:34:40.916945 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:40.907705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:55825 (local address 127.10.109.126:39525)
0620 18:34:40.907776 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:40.907779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:40.908010 (+ 231us) server_negotiation.cc:407] Connection header received
0620 18:34:40.908896 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:40.908897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:40.908908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:40.908965 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:40.909546 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.910034 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.911193 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.911441 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.911514 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:40.913952 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:40.913953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:40.913953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:40.913960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:40.915578 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:40.916006 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:40.916007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:40.916007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:40.916070 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:40.916314 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:40.916315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:40.916316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:40.916433 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:40.916509 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:40.916769 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:34:40.916851 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:34:40.917171 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:40.907690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:39525 (local address 127.10.109.65:55825)
0620 18:34:40.907864 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:40.907897 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:40.908779 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:40.909040 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:40.909043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:40.909288 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:40.909466 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:40.909470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.910136 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.910137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:40.911089 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:40.911093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:40.911545 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:40.911546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:40.911663 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:40.912200 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:40.912208 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:40.913843 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:40.915677 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:40.915684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:40.915686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:40.915914 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:40.916137 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:40.916138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:40.916139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:40.916243 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:40.916501 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:40.916507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:40.916640 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:34:40.916749 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:34:40.916914 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260620 18:34:40.917649 16433 heartbeater.cc:347] Connected to a master server at 127.10.109.126:39525
I20260620 18:34:40.917738 16433 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:40.917923 16433 heartbeater.cc:511] Master 127.10.109.126:39525 requested a full tablet report, sending...
I20260620 18:34:40.918418 16242 ts_manager.cc:194] Registered new tserver with Master: 03d7727594e1459c946c3444311c934a (127.10.109.65:38675)
I20260620 18:34:40.919114 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:55825
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:40.937722 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:41.016530 16438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:41.016760 16438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:41.016796 16438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:41.017918 16438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:41.017985 16438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:41.018033 16438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:41.019479 16438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:41.019723 16438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:41.019944 16438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:41.024808 16444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:41.024973 16438 server_base.cc:1034] running on GCE node
W20260620 18:34:41.024801 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:41.024830 16446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:41.025310 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:41.025550 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:41.026710 16438 hybrid_clock.cc:648] HybridClock initialized: now 1781980481026683 us; error 45 us; skew 500 ppm
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.029115 16438 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:41.029551 16438 webserver.cc:460] Webserver started at http://127.10.109.66:44759/ using document root <none> and password file <none>
I20260620 18:34:41.029819 16438 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:41.029897 16438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:41.030045 16438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:41.031212 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "2242524ab71f4cbf871972b89feac565"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "8d54ea2c1c3e6a803f586c6b7762db68"
server_key_iv: "916eb54d3112a61da26a69fe61656034"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.031579 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "2242524ab71f4cbf871972b89feac565"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "8d54ea2c1c3e6a803f586c6b7762db68"
server_key_iv: "916eb54d3112a61da26a69fe61656034"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.032850 16438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:41.033622 16453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.033836 16438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:41.033926 16438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "2242524ab71f4cbf871972b89feac565"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "8d54ea2c1c3e6a803f586c6b7762db68"
server_key_iv: "916eb54d3112a61da26a69fe61656034"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.034008 16438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:41.052161 16438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:41.053996 16438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:41.054148 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:41.054407 16438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:41.054661 16438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:41.054690 16438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.054710 16438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:41.054723 16438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.060389 16438 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:38203
I20260620 18:34:41.060425 16566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:38203 every 8 connection(s)
I20260620 18:34:41.060709 16438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:41.061981 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16438
I20260620 18:34:41.062074 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:41.062287 10677 external_mini_cluster.cc:1428] Setting key a77ec006361440aa157246415d48f142
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:34:41.070828 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.061371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:46371 (local address 127.10.109.126:39525)
0620 18:34:41.061439 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.061441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.061649 (+ 208us) server_negotiation.cc:407] Connection header received
0620 18:34:41.062816 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.062817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.062829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.062880 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.063568 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.064088 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.065104 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.065436 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.065508 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.067652 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:41.067653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:41.067654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:41.067662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:41.069357 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.069744 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.069745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.069746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.069813 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.070089 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.070089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.070090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.070226 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:41.070298 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:41.070611 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:34:41.070709 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:34:41.071012 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.061336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:39525 (local address 127.10.109.66:46371)
0620 18:34:41.061537 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:41.061570 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:41.062679 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:41.062947 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:41.062951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:41.063295 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:41.063472 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:41.063477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.064198 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.064200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:41.065011 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:41.065013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.065518 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.065519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:41.065610 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.066142 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:41.066171 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:41.067533 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:41.069451 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:41.069456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:41.069457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:41.069664 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:41.069886 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:41.069887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:41.069887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:41.070007 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:41.070321 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:41.070326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:41.070447 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 18:34:41.070588 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:34:41.070781 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260620 18:34:41.071530 16567 heartbeater.cc:347] Connected to a master server at 127.10.109.126:39525
I20260620 18:34:41.071642 16567 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:41.071822 16567 heartbeater.cc:511] Master 127.10.109.126:39525 requested a full tablet report, sending...
I20260620 18:34:41.072211 16242 ts_manager.cc:194] Registered new tserver with Master: 2242524ab71f4cbf871972b89feac565 (127.10.109.66:38203)
I20260620 18:34:41.072590 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:46371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:41.089162 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:41.168169 16572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:41.168355 16572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:41.168378 16572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:41.169427 16572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:41.169462 16572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:41.169487 16572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:41.170848 16572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:41.171060 16572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:41.171260 16572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:41.176554 16580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:41.176549 16578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:41.176550 16577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:41.176834 16572 server_base.cc:1034] running on GCE node
I20260620 18:34:41.177093 16572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:41.177330 16572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:41.178491 16572 hybrid_clock.cc:648] HybridClock initialized: now 1781980481178480 us; error 30 us; skew 500 ppm
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.181021 16572 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:41.181546 16572 webserver.cc:460] Webserver started at http://127.10.109.67:34927/ using document root <none> and password file <none>
I20260620 18:34:41.181771 16572 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:41.181836 16572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:41.181995 16572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:41.183164 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "f60f97c5ed804520b8f0e7db3e77586e"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "6b60099dfd327000f1532805766cf9b3"
server_key_iv: "679d87c9b3e41a8cdf9a4dff0fa2db5c"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.183519 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "f60f97c5ed804520b8f0e7db3e77586e"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "6b60099dfd327000f1532805766cf9b3"
server_key_iv: "679d87c9b3e41a8cdf9a4dff0fa2db5c"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.184793 16572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:41.185540 16587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.185714 16572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:41.185809 16572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "f60f97c5ed804520b8f0e7db3e77586e"
format_stamp: "Formatted at 2026-06-20 18:34:41 on dist-test-slave-3jql"
server_key: "6b60099dfd327000f1532805766cf9b3"
server_key_iv: "679d87c9b3e41a8cdf9a4dff0fa2db5c"
server_key_version: "encryptionkey@0"
I20260620 18:34:41.185890 16572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:41.195874 16572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:41.197690 16572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:41.197850 16572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:41.198078 16572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:41.198385 16572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:41.198438 16572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.198474 16572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:41.198504 16572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:41.204651 16572 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:32889
I20260620 18:34:41.204691 16700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:32889 every 8 connection(s)
I20260620 18:34:41.205055 16572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:41.213644 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16572
I20260620 18:34:41.213754 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:41.213972 10677 external_mini_cluster.cc:1428] Setting key 414a23b7d7185a2adb79022f5c46d399
I20260620 18:34:41.214576 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.205800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:56915 (local address 127.10.109.126:39525)
0620 18:34:41.205871 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.205874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.206102 (+ 228us) server_negotiation.cc:407] Connection header received
0620 18:34:41.207058 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.207060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.207070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.207138 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.207760 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.208260 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.209252 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.209430 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.209487 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.211634 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:41.211636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:41.211636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:41.211643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:41.213043 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.213473 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.213474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.213474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.213538 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.213867 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.213869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.213869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.213979 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:41.214062 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:41.214409 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 18:34:41.214486 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:34:41.214797 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.205787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:39525 (local address 127.10.109.67:56915)
0620 18:34:41.205939 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:41.205990 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:41.206941 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:41.207191 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:41.207195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:41.207478 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:41.207667 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:41.207671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.208339 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.208340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:41.209156 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:41.209159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.209524 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.209525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:41.209648 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.210127 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:41.210135 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:41.211548 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:41.213142 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:41.213147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:41.213149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:41.213380 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:41.213607 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:41.213607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:41.213608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:41.213705 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:41.214052 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:41.214058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:41.214253 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 18:34:41.214373 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:34:41.214527 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260620 18:34:41.215220 16701 heartbeater.cc:347] Connected to a master server at 127.10.109.126:39525
I20260620 18:34:41.215297 16701 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:41.215479 16701 heartbeater.cc:511] Master 127.10.109.126:39525 requested a full tablet report, sending...
I20260620 18:34:41.215831 16242 ts_manager.cc:194] Registered new tserver with Master: f60f97c5ed804520b8f0e7db3e77586e (127.10.109.67:32889)
I20260620 18:34:41.216229 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:56915
I20260620 18:34:41.225795 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/test-admin.keytab.
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.240711 10677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:34:41.242357 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:41.241512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.10.109.65:38675)
0620 18:34:41.241697 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.241702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.241711 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:41.241738 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.241740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.241752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.241817 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.242264 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51774: BlockingRecv error: recv got EOF from 127.0.0.1:51774 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
W20260620 18:34:41.242367 10677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.243440 10677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.244342 10677 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:41.252483 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.244838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.10.109.65:38675)
0620 18:34:41.244903 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.244905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.244947 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:34:41.245025 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.245026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.245038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.245110 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.245501 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.246213 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.247001 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.247324 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.247415 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.249441 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:41.249456 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:41.249456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:41.249463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:41.251290 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.251666 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.251669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.251669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.251726 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.251926 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.251927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.251928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.252060 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:41.252217 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:41.252289 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:34:41.252366 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 18:34:41.254228 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:41.253725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.10.109.65:38675)
0620 18:34:41.253758 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.253759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.253807 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:34:41.253875 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.253876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.253884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.253929 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.254165 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51796: BlockingRecv error: recv got EOF from 127.0.0.1:51796 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 18:34:41.254351 10677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.255476 10677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.256409 10677 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:41.263411 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.256884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.10.109.65:38675)
0620 18:34:41.256936 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.256938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.256970 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:34:41.257024 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.257025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.257032 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.257078 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.257442 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.257901 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.258661 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.258915 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.258993 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.261021 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:41.261022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:41.261022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:41.261028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:41.262372 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.262683 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.262683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.262684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.262731 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.262977 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.262978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.262978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.263067 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:41.263139 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:41.263245 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:34:41.263299 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 18:34:41.265055 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:41.264425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.10.109.65:38675)
0620 18:34:41.264488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.264491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.264609 (+ 118us) server_negotiation.cc:407] Connection header received
0620 18:34:41.264679 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.264679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.264689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.264759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.265019 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51806: BlockingRecv error: recv got EOF from 127.0.0.1:51806 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 18:34:41.265215 10677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980452801806-10677-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.266199 10677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:41.267153 10677 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:34:41.274123 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:41.267633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.10.109.65:38675)
0620 18:34:41.267665 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:41.267667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:41.267758 (+ 91us) server_negotiation.cc:407] Connection header received
0620 18:34:41.267819 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:41.267819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:41.267828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:41.267876 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:41.268228 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.268669 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.269697 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:41.270022 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:41.270085 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:41.271969 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:41.271970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:41.271970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:41.271976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:41.273107 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.273465 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.273466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.273466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.273531 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:41.273707 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:41.273708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:41.273708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:41.273790 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:41.273851 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:41.274017 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:34:41.274071 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 18:34:41.274799 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16304
I20260620 18:34:41.279414 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16438
I20260620 18:34:41.283595 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16572
I20260620 18:34:41.287833 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16210
2026-06-20T18:34:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1933 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16721](info): set up 2 sockets
Jun 20 18:34:41 dist-test-slave-3jql krb5kdc[16721](info): commencing operation
krb5kdc: starting...
W20260620 18:34:42.315920 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab.
Entry for principal oryx/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab.
I20260620 18:34:42.442879 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43923
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab
--principal=oryx/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:42.521646 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:42.521868 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:42.521903 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:42.523036 16734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:42.523100 16734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:42.523125 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:42.523159 16734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:42.523183 16734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:42.524607 16734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab
--principal=oryx/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:42.524859 16734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:42.525071 16734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:42.529981 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.530004 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.530071 16734 server_base.cc:1034] running on GCE node
W20260620 18:34:42.529982 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.530470 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:42.530786 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:42.531949 16734 hybrid_clock.cc:648] HybridClock initialized: now 1781980482531933 us; error 47 us; skew 500 ppm
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:42.534291 16734 init.cc:377] Logged in from keytab as oryx/127.10.109.126@KRBTEST.COM (short username oryx)
I20260620 18:34:42.534721 16734 webserver.cc:460] Webserver started at http://127.10.109.126:45809/ using document root <none> and password file <none>
I20260620 18:34:42.535000 16734 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:42.535046 16734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:42.535185 16734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:42.536212 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "1a673faa5b894849b1cfe935680c6f45"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "af02c3775cddac03f983e07cb70c3716"
server_key_iv: "792178190c871479db28c504706ca995"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.536609 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "1a673faa5b894849b1cfe935680c6f45"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "af02c3775cddac03f983e07cb70c3716"
server_key_iv: "792178190c871479db28c504706ca995"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.538071 16734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:42.538847 16749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.539000 16734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.539084 16734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "1a673faa5b894849b1cfe935680c6f45"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "af02c3775cddac03f983e07cb70c3716"
server_key_iv: "792178190c871479db28c504706ca995"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.539135 16734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:42.553723 16734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:42.555835 16734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:42.555969 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:42.560014 16734 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:43923
I20260620 18:34:42.560091 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:43923 every 8 connection(s)
I20260620 18:34:42.560459 16734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:42.561476 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:42.563640 16802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Bootstrap starting.
I20260620 18:34:42.564194 16802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:42.564486 16802 log.cc:824] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:42.565177 16802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: No bootstrap required, opened a new log
I20260620 18:34:42.566733 16802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } }
I20260620 18:34:42.566920 16802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:42.566969 16802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a673faa5b894849b1cfe935680c6f45, State: Initialized, Role: FOLLOWER
I20260620 18:34:42.567113 16802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } }
I20260620 18:34:42.567193 16802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:42.567241 16802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:42.567292 16802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:42.567507 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16734
I20260620 18:34:42.567605 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:42.567757 10677 external_mini_cluster.cc:1428] Setting key 8528e95d76f78629d3a9ca569d261d3c
I20260620 18:34:42.568182 16802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } }
I20260620 18:34:42.568315 16802 leader_election.cc:302] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a673faa5b894849b1cfe935680c6f45; no voters:
I20260620 18:34:42.568461 16802 leader_election.cc:288] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:42.568607 16807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:42.568683 16802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:42.568776 16807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: Becoming Leader. State: Replica: 1a673faa5b894849b1cfe935680c6f45, State: Running, Role: LEADER
I20260620 18:34:42.568899 16807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } }
I20260620 18:34:42.569252 16807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a673faa5b894849b1cfe935680c6f45. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } }
I20260620 18:34:42.569260 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } }
I20260620 18:34:42.569377 16807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:42.569486 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:42.569815 16816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:42.570348 16816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:42.571833 16824 mvcc.cc:205] Tried to move back new op lower bound from 7298992056612106240 to 7298992056602583040. Current Snapshot: MvccSnapshot[applied={T|T < 7298992056612106240}]
I20260620 18:34:42.573760 16816 catalog_manager.cc:1347] Generated new cluster ID: a335d32e3b494575aaee5e639a2be3e6
I20260620 18:34:42.573827 16816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:42.580044 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.568286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.10.109.126:43923)
0620 18:34:42.568474 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:42.568483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:42.568520 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:34:42.568634 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:42.568640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:42.568691 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:42.569689 (+ 998us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:42.569814 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.571349 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.572398 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.573007 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.573118 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.575932 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:42.575937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:42.575937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:42.575948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:42.578428 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.578841 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.578844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.578844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.578960 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.579161 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.579161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.579162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.579465 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:42.579619 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:42.579695 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:34:42.579838 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260620 18:34:42.586077 16816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:42.586817 16816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:42.590335 16816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Generated new TSK 0
I20260620 18:34:42.590519 16816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
I20260620 18:34:42.625280 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:42.706007 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:42.706198 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:42.706220 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:42.707413 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:42.707453 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:42.707479 16828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:42.708838 16828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:42.709052 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:42.709255 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:42.714046 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.714032 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.714011 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.714435 16828 server_base.cc:1034] running on GCE node
I20260620 18:34:42.714640 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:42.714888 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:42.716078 16828 hybrid_clock.cc:648] HybridClock initialized: now 1781980482716064 us; error 25 us; skew 500 ppm
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:42.718564 16828 init.cc:377] Logged in from keytab as oryx/127.10.109.65@KRBTEST.COM (short username oryx)
I20260620 18:34:42.719084 16828 webserver.cc:460] Webserver started at http://127.10.109.65:40811/ using document root <none> and password file <none>
I20260620 18:34:42.719326 16828 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:42.719394 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:42.719545 16828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:42.720530 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "f391411f1fe24c91864a8e92daf750fd"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "e1f107748acb491cd724f005ccdfa4b6"
server_key_iv: "343d286c9bc04402a88730949cdc0487"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.720885 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "f391411f1fe24c91864a8e92daf750fd"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "e1f107748acb491cd724f005ccdfa4b6"
server_key_iv: "343d286c9bc04402a88730949cdc0487"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.722286 16828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:42.723178 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.723361 16828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:42.723464 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "f391411f1fe24c91864a8e92daf750fd"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "e1f107748acb491cd724f005ccdfa4b6"
server_key_iv: "343d286c9bc04402a88730949cdc0487"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.723544 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:42.733822 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:42.736022 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:42.736174 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:42.736383 16828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:42.736671 16828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:42.736725 16828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.736764 16828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:42.736791 16828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.742774 16828 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:33709
I20260620 18:34:42.742880 16956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:33709 every 8 connection(s)
I20260620 18:34:42.743166 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:42.749887 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16828
I20260620 18:34:42.750011 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:42.750248 10677 external_mini_cluster.cc:1428] Setting key cbdb2d5ea0e16336fd0eda2fe6f58e9c
I20260620 18:34:42.753166 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.743845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:34673 (local address 127.10.109.126:43923)
0620 18:34:42.743908 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:42.743910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:42.744124 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:34:42.744639 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:42.744641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:42.744653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:42.744707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:42.745632 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.746145 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.747210 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.747412 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.747488 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.750036 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:42.750038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:42.750038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:42.750046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:42.751747 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.752216 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.752217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.752217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.752277 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.752534 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.752535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.752536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.752653 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:42.752729 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:42.752997 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:34:42.753073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:42.753378 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.743845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:34673)
0620 18:34:42.744008 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:42.744040 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:42.744532 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:42.744784 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:42.744788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:42.745351 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:42.745529 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:42.745534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.746242 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.746243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:42.747111 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:42.747114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.747495 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.747496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:42.747613 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.748054 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:42.748062 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:42.749909 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:42.751890 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:42.751898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:42.751900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:42.752127 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:42.752353 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:42.752354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:42.752355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:42.752461 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:42.752721 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:42.752728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:42.752865 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 18:34:42.752976 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:34:42.753134 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 18:34:42.753876 16957 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:42.753980 16957 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:42.754168 16957 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:42.754674 16766 ts_manager.cc:194] Registered new tserver with Master: f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709)
I20260620 18:34:42.755291 16766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:34673
WARNING: no policy specified for oryx/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
I20260620 18:34:42.775926 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:42.854624 16962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:42.854878 16962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:42.854905 16962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:42.855966 16962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:42.856002 16962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:42.856027 16962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:42.857343 16962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:42.857522 16962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:42.857678 16962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:42.862682 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:42.862690 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.862892 16962 server_base.cc:1034] running on GCE node
W20260620 18:34:42.863050 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:42.863268 16962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:42.863456 16962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:42.864590 16962 hybrid_clock.cc:648] HybridClock initialized: now 1781980482864580 us; error 26 us; skew 500 ppm
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:42.867004 16962 init.cc:377] Logged in from keytab as oryx/127.10.109.66@KRBTEST.COM (short username oryx)
I20260620 18:34:42.867455 16962 webserver.cc:460] Webserver started at http://127.10.109.66:43803/ using document root <none> and password file <none>
I20260620 18:34:42.867678 16962 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:42.867745 16962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:42.867895 16962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:42.868923 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "4560515d11024407b702f9acb1e19eff"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "ab35b4732fde58c6d0707fa6f53c4c08"
server_key_iv: "e4ee67c78846cd281975986542a7fbc1"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.869303 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "4560515d11024407b702f9acb1e19eff"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "ab35b4732fde58c6d0707fa6f53c4c08"
server_key_iv: "e4ee67c78846cd281975986542a7fbc1"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.870713 16962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:42.871475 16977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.871649 16962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:42.871742 16962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "4560515d11024407b702f9acb1e19eff"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "ab35b4732fde58c6d0707fa6f53c4c08"
server_key_iv: "e4ee67c78846cd281975986542a7fbc1"
server_key_version: "encryptionkey@0"
I20260620 18:34:42.871827 16962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:42.880954 16962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:42.883157 16962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:42.883308 16962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:42.883515 16962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:42.883792 16962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:42.883844 16962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.883885 16962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:42.883940 16962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:42.889781 16962 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37641
I20260620 18:34:42.889847 17090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37641 every 8 connection(s)
I20260620 18:34:42.890193 16962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:42 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:42.899974 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.891025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:60855 (local address 127.10.109.126:43923)
0620 18:34:42.891104 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:42.891107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:42.891300 (+ 193us) server_negotiation.cc:407] Connection header received
0620 18:34:42.891865 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:42.891867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:42.891880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:42.891939 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:42.892992 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.893550 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.894644 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.894940 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.895034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.897174 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:42.897175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:42.897176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:42.897182 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:42.898502 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.899044 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.899046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.899046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.899107 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:42.899350 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:42.899351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:42.899351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:42.899465 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:42.899542 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:42.899800 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:34:42.899880 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:34:42.900167 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:42.891002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:60855)
0620 18:34:42.891171 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:42.891208 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:42.891748 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:42.892030 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:42.892033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:42.892621 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:42.892818 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:42.892823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.893653 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.893655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:42.894536 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:42.894539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:42.895044 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:42.895046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:42.895174 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:42.895619 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:42.895627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:42.897057 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:42.898616 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:42.898623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:42.898625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:42.898951 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:42.899175 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:42.899176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:42.899176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:42.899278 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:42.899533 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:42.899539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:42.899671 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 18:34:42.899787 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:34:42.899939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260620 18:34:42.900254 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 16962
I20260620 18:34:42.900379 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:42.900571 10677 external_mini_cluster.cc:1428] Setting key 811f9e5905f472ecfa5a558cdf166622
I20260620 18:34:42.900724 17091 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:42.900820 17091 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:42.900992 17091 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:42.901391 16766 ts_manager.cc:194] Registered new tserver with Master: 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:42.901821 16766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:60855
WARNING: no policy specified for oryx/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
I20260620 18:34:42.925913 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:43.007696 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.007943 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.007982 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.009151 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.009219 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.009265 17096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:43.010769 17096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:43.011051 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.011289 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.016253 17101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.016240 17104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.016228 17102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.016630 17096 server_base.cc:1034] running on GCE node
I20260620 18:34:43.016849 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.017113 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.018280 17096 hybrid_clock.cc:648] HybridClock initialized: now 1781980483018262 us; error 45 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.020962 17096 init.cc:377] Logged in from keytab as oryx/127.10.109.67@KRBTEST.COM (short username oryx)
I20260620 18:34:43.021325 17096 webserver.cc:460] Webserver started at http://127.10.109.67:41905/ using document root <none> and password file <none>
I20260620 18:34:43.021523 17096 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.021564 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.021656 17096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:43.022612 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "9614cbfa16733c21eddbaad15e65a682"
server_key_iv: "3065a922f15ac1ef006b56dbbf2480d2"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.022974 17096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "9614cbfa16733c21eddbaad15e65a682"
server_key_iv: "3065a922f15ac1ef006b56dbbf2480d2"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.024176 17096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:43.024977 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.025223 17096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:43.025305 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "9614cbfa16733c21eddbaad15e65a682"
server_key_iv: "3065a922f15ac1ef006b56dbbf2480d2"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.025430 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.053654 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.055992 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:43.056140 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.056396 17096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:43.056713 17096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:43.056766 17096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.056820 17096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:43.056847 17096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.062562 17096 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:34819
I20260620 18:34:43.062638 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:34819 every 8 connection(s)
I20260620 18:34:43.063014 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:43.071221 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17096
I20260620 18:34:43.071365 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:43.071564 10677 external_mini_cluster.cc:1428] Setting key bc3ee1d03c59160bc7f180fb744f8ca8
I20260620 18:34:43.072256 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.063701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:48583 (local address 127.10.109.126:43923)
0620 18:34:43.063774 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.063777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.063958 (+ 181us) server_negotiation.cc:407] Connection header received
0620 18:34:43.064429 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.064430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.064440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.064502 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.065381 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.065921 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.066910 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.067096 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.067158 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.069251 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.069252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.069252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.069258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.070721 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.071193 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.071194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.071195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.071257 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.071500 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.071501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.071501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.071619 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.071700 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.072064 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.072154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:34:43.072481 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.063681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:48583)
0620 18:34:43.063842 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.063875 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.064323 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.064551 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.064554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.065102 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.065278 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.065283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.066008 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.066009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.066836 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.066838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.067173 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.067174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.067275 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.067751 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.067758 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.069151 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.070867 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.070874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.070876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.071097 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.071330 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.071331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.071332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.071430 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.071705 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.071712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.071912 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.072032 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.072232 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260620 18:34:43.072844 17225 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:43.072935 17225 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.073069 17225 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:43.073416 16766 ts_manager.cc:194] Registered new tserver with Master: f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:43.073737 16766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:48583
I20260620 18:34:43.083390 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.109.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.125.keytab.
Entry for principal oryx/127.10.109.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.125.keytab.
I20260620 18:34:43.109349 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:45667
--webserver_interface=127.10.109.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.125.keytab
--principal=oryx/127.10.109.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43923,127.10.109.125:45667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:43.187634 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.187862 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.187899 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.189009 17230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:43.189072 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.189096 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.189124 17230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:43.189147 17230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:43.190552 17230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43923,127.10.109.125:45667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.125.keytab
--principal=oryx/127.10.109.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:45667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.109.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
W20260620 18:34:43.190677 17230 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.109.126:43923,127.10.109.125:45667'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:34:43.190850 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.191072 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.195607 17236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.195607 17235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.195674 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.195796 17230 server_base.cc:1034] running on GCE node
I20260620 18:34:43.196059 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.196287 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.197434 17230 hybrid_clock.cc:648] HybridClock initialized: now 1781980483197423 us; error 28 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.200031 17230 init.cc:377] Logged in from keytab as oryx/127.10.109.125@KRBTEST.COM (short username oryx)
I20260620 18:34:43.200481 17230 webserver.cc:460] Webserver started at http://127.10.109.125:46533/ using document root <none> and password file <none>
I20260620 18:34:43.200714 17230 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.200757 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.200897 17230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:43.201901 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/instance:
uuid: "5faca9ffaeb1442589a58040c0b33b04"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "8f1110b052bd75c9f2ad90de05093f50"
server_key_iv: "897866adbd4455a3964a8d6bf6644c82"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.202248 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal/instance:
uuid: "5faca9ffaeb1442589a58040c0b33b04"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "8f1110b052bd75c9f2ad90de05093f50"
server_key_iv: "897866adbd4455a3964a8d6bf6644c82"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.203558 17230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:34:43.204277 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.204479 17230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:43.204547 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "5faca9ffaeb1442589a58040c0b33b04"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "8f1110b052bd75c9f2ad90de05093f50"
server_key_iv: "897866adbd4455a3964a8d6bf6644c82"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.204638 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.222486 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.224725 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:43.224869 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.229058 17230 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.125:45667
I20260620 18:34:43.229094 17297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.125:45667 every 8 connection(s)
I20260620 18:34:43.229496 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
I20260620 18:34:43.230496 17298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:43.231987 17298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:34:43.233660 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17230
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.125@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
I20260620 18:34:43.241571 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.232681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.10.109.126:43923)
0620 18:34:43.232750 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.232753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.233003 (+ 250us) server_negotiation.cc:407] Connection header received
0620 18:34:43.233438 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.233439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.233449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.233515 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.234368 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.234967 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.235971 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.236182 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.236232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.238624 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.238626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.238626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.238634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.240148 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.240563 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.240564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.240564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.240627 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.240873 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.240874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.240875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.241053 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.241129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.241404 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.241483 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:34:43.241737 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.232660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39608)
0620 18:34:43.232853 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.232912 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.233328 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.233600 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.233604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.233975 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.234256 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.234263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.235076 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.235077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.235870 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.235872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.236258 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.236259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.236389 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.236930 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.236937 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.238529 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.240241 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.240248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.240250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.240485 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.240702 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.240703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.240703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.240802 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.241130 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.241136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.241264 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.241371 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.241514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260620 18:34:43.242357 17298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:34:43.242659 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.234141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.10.109.125:45667)
0620 18:34:43.234284 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.234287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.234346 (+ 59us) server_negotiation.cc:407] Connection header received
0620 18:34:43.234389 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.234391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.234411 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.234487 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.235134 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.235881 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.236779 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.237078 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.237161 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.239757 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.239762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.239763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.239768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.241328 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.241714 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.241718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.241718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.241783 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.241993 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.241994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.241994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.242160 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.242302 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.242457 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.242541 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.125@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
I20260620 18:34:43.248966 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.242613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45262)
0620 18:34:43.242652 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.242662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.242697 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.243022 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.243023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.243161 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.243215 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.243217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.243867 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.243867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.244321 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.244323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.244707 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.244707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.244766 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.245082 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.245085 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.246146 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.248084 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.248085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.248085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.248244 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.248422 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.248422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.248423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.248471 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.248691 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.248692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.248735 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.248812 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.248887 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 18:34:43.249087 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.242639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.10.109.125:45667)
0620 18:34:43.242785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.242787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.242794 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:34:43.242937 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.242938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.242947 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.243016 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.243275 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.243725 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.244409 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.244624 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.244707 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.246218 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.246219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.246219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.246224 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.248007 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.248308 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.248308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.248309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.248357 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.248523 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.248523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.248524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.248611 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.248660 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.248827 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.248931 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260620 18:34:43.250339 17298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Bootstrap starting.
I20260620 18:34:43.251019 17298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:43.251293 17298 log.cc:824] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:43.252010 17298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: No bootstrap required, opened a new log
I20260620 18:34:43.253377 17298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } }
I20260620 18:34:43.253494 17298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:43.253515 17298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faca9ffaeb1442589a58040c0b33b04, State: Initialized, Role: FOLLOWER
I20260620 18:34:43.253597 17298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } }
I20260620 18:34:43.253931 17298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:43.253897 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } }
I20260620 18:34:43.254014 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:34:43.255668 17316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:43.255769 17316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:43.257148 17230 master_runner.cc:428] Detected that this master 5faca9ffaeb1442589a58040c0b33b04 is joining an existing cluster
I20260620 18:34:43.257220 17230 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.109.125:45667
I20260620 18:34:43.265168 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.258105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39618)
0620 18:34:43.258265 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.258277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.258360 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.258544 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.258545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.258784 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.258868 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.258870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.259657 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.259658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.260553 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.260555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.261062 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.261063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.261129 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.261681 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.261686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.262138 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.263962 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.263963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.263963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.264206 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.264424 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.264425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.264426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.264498 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.264836 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.264836 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.264881 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.264968 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.265068 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260620 18:34:43.265163 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.258196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.10.109.126:43923)
0620 18:34:43.258250 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.258252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.258334 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:34:43.258449 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.258450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.258464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.258531 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.258958 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.259577 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.260643 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.260940 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.261010 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.262235 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.262237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.262238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.262245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.263899 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.264297 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.264298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.264298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.264351 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.264579 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.264580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.264580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.264751 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.264847 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.265010 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.265078 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:34:43.265738 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.258309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45274)
0620 18:34:43.258502 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.258511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.258568 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.258689 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.258689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.258892 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.258971 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.258974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.259851 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.259852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.260649 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.260651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.261005 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.261006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.261062 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.261495 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.261500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.261921 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.263897 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.263898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.263899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.264206 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.264885 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.264886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.264886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.264947 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.265215 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.265216 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.265454 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.265546 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.265637 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":59,"threads_started":1}
I20260620 18:34:43.265915 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.258430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.10.109.125:45667)
0620 18:34:43.258459 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.258461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.258550 (+ 89us) server_negotiation.cc:407] Connection header received
0620 18:34:43.258615 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.258615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.258624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.258665 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.259051 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.259770 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.260718 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.260918 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.260993 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.262020 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.262022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.262022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.262028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.263804 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.264441 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.264442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.264442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.264502 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.265021 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.265021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.265022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.265136 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.265176 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.265767 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.265848 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.126@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
I20260620 18:34:43.275357 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.267531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.10.109.125:45667)
0620 18:34:43.267567 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.267569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.267695 (+ 126us) server_negotiation.cc:407] Connection header received
0620 18:34:43.267787 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.267788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.267798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.267852 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.268339 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.268810 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.269800 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.269992 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.270071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.272262 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.272263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.272264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.272270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.273504 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.273946 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.273947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.273947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.274003 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.274208 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.274209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.274209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.274308 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.274388 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.274960 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.275262 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:43.275477 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.267479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45280)
0620 18:34:43.267590 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.267626 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.267725 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.267859 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.267862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.268211 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.268254 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.268256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.268883 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.268884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.269689 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.269691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.270076 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.270077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.270150 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.270607 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.270611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.272174 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.273602 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.273608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.273610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.273808 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.274081 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.274082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.274082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.274146 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.274380 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.274387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.274436 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.274899 (+ 463us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.275333 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260620 18:34:43.276082 16766 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.109.125:45667: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04"
member_type: NON_VOTER
last_known_addr {
host: "127.10.109.125"
port: 45667
}
attrs {
promote: true
}
}
dest_uuid: "1a673faa5b894849b1cfe935680c6f45"
cas_config_opid_index: -1
I20260620 18:34:43.276410 16766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }
I20260620 18:34:43.276917 17326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.277300 17272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:43.278071 17272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:34:43.278208 17326 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.109.125:45667
I20260620 18:34:43.278465 17230 master.cc:537] Master@127.10.109.125:45667 shutting down...
I20260620 18:34:43.278479 17326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:43.278684 17325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.278825 17325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:43.279170 16751 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 18:34:43.281416 17230 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:43.281504 17230 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:43.281551 17230 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: stopping tablet replica
I20260620 18:34:43.287174 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.287042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45294)
0620 18:34:43.287113 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.287149 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:45667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:34:43.287398 16751 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 -> Peer 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125:45667): Couldn't send request to peer 5faca9ffaeb1442589a58040c0b33b04. Status: Network error: Client connection negotiation failed: client connection to 127.10.109.125:45667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:34:43.294292 17230 master.cc:552] Master@127.10.109.125:45667 shutdown complete.
I20260620 18:34:43.294335 17230 master_runner.cc:305] Clearing existing system tablet
I20260620 18:34:43.294497 17230 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.294538 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.295311 17230 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.295828 17330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.295984 17230 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.296051 17230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:43.296083 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "5faca9ffaeb1442589a58040c0b33b04"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "8f1110b052bd75c9f2ad90de05093f50"
server_key_iv: "897866adbd4455a3964a8d6bf6644c82"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.297219 17230 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:43.298715 17230 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:34:43.298882 17230 log.cc:1196] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:34:43.299088 17230 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Deleting consensus metadata
I20260620 18:34:43.299260 17230 master_runner.cc:315] Copying system tablet from 127.10.109.126:43923
I20260620 18:34:43.299916 17230 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: tablet copy: Beginning tablet copy session from remote peer at address 127.10.109.126:43923
I20260620 18:34:43.306381 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.300277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.10.109.126:43923)
0620 18:34:43.300331 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.300333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.300395 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:34:43.300485 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.300486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.300498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.300561 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.300919 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.301426 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.302180 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.302396 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.302490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.303811 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.303813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.303813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.303821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.305204 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.305596 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.305597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.305598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.305652 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.305862 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.305863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.305863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.305981 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.306064 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.306213 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.306288 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:34:43.306401 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.300205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39620)
0620 18:34:43.300328 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.300342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.300415 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.300566 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.300567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.300757 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.300814 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.300815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.301515 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.301516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.302070 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.302072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.302490 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.302491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.302563 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.303134 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.303142 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.303696 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.305325 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.305326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.305326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.305519 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.305731 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.305732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.305732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.305798 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.306055 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.306056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.306101 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.306189 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.306292 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260620 18:34:43.306960 16786 tablet_copy_service.cc:140] P 1a673faa5b894849b1cfe935680c6f45: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5faca9ffaeb1442589a58040c0b33b04 ({username='oryx', principal='oryx/127.10.109.125@KRBTEST.COM'} at 127.0.0.1:39620)
I20260620 18:34:43.307117 16786 tablet_copy_service.cc:161] P 1a673faa5b894849b1cfe935680c6f45: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5faca9ffaeb1442589a58040c0b33b04 at {username='oryx', principal='oryx/127.10.109.125@KRBTEST.COM'} at 127.0.0.1:39620: session id = 5faca9ffaeb1442589a58040c0b33b04-00000000000000000000000000000000
I20260620 18:34:43.307731 16786 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:34:43.308336 17230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:34:43.309516 17230 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: tablet copy: Starting download of 0 data blocks...
I20260620 18:34:43.309670 17230 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: tablet copy: Starting download of 1 WAL segments...
I20260620 18:34:43.310519 17230 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:34:43.311733 16786 tablet_copy_service.cc:342] P 1a673faa5b894849b1cfe935680c6f45: Request end of tablet copy session 5faca9ffaeb1442589a58040c0b33b04-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.109.125@KRBTEST.COM'} at 127.0.0.1:39620
I20260620 18:34:43.311795 16786 tablet_copy_service.cc:433] P 1a673faa5b894849b1cfe935680c6f45: ending tablet copy session 5faca9ffaeb1442589a58040c0b33b04-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5faca9ffaeb1442589a58040c0b33b04
I20260620 18:34:43.314112 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.315351 17337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.315431 17338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.315588 17340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.318921 17230 server_base.cc:1034] running on GCE node
I20260620 18:34:43.319018 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.319199 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.320338 17230 hybrid_clock.cc:648] HybridClock initialized: now 1781980483320325 us; error 31 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.321751 17230 init.cc:377] Logged in from keytab as oryx/127.10.109.125@KRBTEST.COM (short username oryx)
I20260620 18:34:43.322043 17230 webserver.cc:460] Webserver started at http://127.10.109.125:33347/ using document root <none> and password file <none>
I20260620 18:34:43.322162 17230 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.322219 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.322890 17230 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:43.323535 17347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.323891 17230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:43.323961 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "5faca9ffaeb1442589a58040c0b33b04"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "8f1110b052bd75c9f2ad90de05093f50"
server_key_iv: "897866adbd4455a3964a8d6bf6644c82"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.324023 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.329533 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.331696 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.334842 17230 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.125:45667
I20260620 18:34:43.335132 17399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.125:45667 every 8 connection(s)
I20260620 18:34:43.335294 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
I20260620 18:34:43.336069 17400 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:34:43.336499 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Bootstrap starting.
I20260620 18:34:43.338555 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:34:43.338809 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: Bootstrap complete.
I20260620 18:34:43.339013 17400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }
I20260620 18:34:43.339079 17400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5faca9ffaeb1442589a58040c0b33b04, State: Initialized, Role: LEARNER
I20260620 18:34:43.339105 17400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }
I20260620 18:34:43.339300 17400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:43.339354 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.339466 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:34:43.340760 17414 catalog_manager.cc:1259] Loaded cluster ID: a335d32e3b494575aaee5e639a2be3e6
I20260620 18:34:43.340822 17414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: loading cluster ID for follower catalog manager: success
I20260620 18:34:43.341876 17414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: acquiring CA information for follower catalog manager: success
I20260620 18:34:43.342133 17414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:34:43.351200 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.344251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.10.109.125:45667)
0620 18:34:43.344358 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.344360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.344376 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:34:43.344398 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.344399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.344409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.344481 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.345178 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.345796 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.346902 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.347125 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.347186 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.348581 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.348582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.348582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.348588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.350064 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.350452 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.350453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.350454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.350501 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.350702 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.350703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.350703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.350840 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.350935 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.351073 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.351132 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.10.109.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.124.keytab.
Entry for principal oryx/127.10.109.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.124.keytab.
I20260620 18:34:43.381768 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:42029
--webserver_interface=127.10.109.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.124.keytab
--principal=oryx/127.10.109.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:43.472551 17418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.472749 17418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.472771 17418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.473841 17418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:43.473874 17418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.473888 17418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.473901 17418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:43.473913 17418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:43.475445 17418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.124.keytab
--principal=oryx/127.10.109.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:42029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.109.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:43.475690 17418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.475868 17418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:34:43.480700 17418 server_base.cc:1034] running on GCE node
W20260620 18:34:43.480671 17426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.480775 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.480662 17424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.481089 17418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.481362 17418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.482524 17418 hybrid_clock.cc:648] HybridClock initialized: now 1781980483482520 us; error 37 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.485013 17418 init.cc:377] Logged in from keytab as oryx/127.10.109.124@KRBTEST.COM (short username oryx)
I20260620 18:34:43.485450 17418 webserver.cc:460] Webserver started at http://127.10.109.124:39301/ using document root <none> and password file <none>
I20260620 18:34:43.485678 17418 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.485745 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.485877 17418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:43.486970 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/instance:
uuid: "93d350436b7d4ff48289c2db754a4e72"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "672c076fb24c63f9bd99c55341a62059"
server_key_iv: "6d09d6049171e6796141075ac9e86f0b"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.487329 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal/instance:
uuid: "93d350436b7d4ff48289c2db754a4e72"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "672c076fb24c63f9bd99c55341a62059"
server_key_iv: "6d09d6049171e6796141075ac9e86f0b"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.488600 17418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.489360 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.489570 17418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.489666 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "93d350436b7d4ff48289c2db754a4e72"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "672c076fb24c63f9bd99c55341a62059"
server_key_iv: "6d09d6049171e6796141075ac9e86f0b"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.489748 17418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.503852 17418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.505602 17418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:43.505756 17418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.509569 17418 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.124:42029
I20260620 18:34:43.509598 17485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.124:42029 every 8 connection(s)
I20260620 18:34:43.509970 17418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
I20260620 18:34:43.510988 17486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:43.512462 17486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:34:43.516875 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17418
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.124@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:43.522830 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.513293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.10.109.126:43923)
0620 18:34:43.513353 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.513355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.513432 (+ 77us) server_negotiation.cc:407] Connection header received
0620 18:34:43.513926 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.513928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.513938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.513998 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.515335 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.515827 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.516866 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.517172 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.517242 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.519727 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.519729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.519729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.519736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.521298 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.521809 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.521810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.521810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.521874 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.522158 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.522159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.522159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.522293 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.522361 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.522640 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.522716 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:43.523001 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.513119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39628)
0620 18:34:43.513289 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.513344 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.513775 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.514253 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.514260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.514996 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.515220 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.515227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.515962 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.515963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.516752 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.516754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.517292 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.517294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.517473 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.518103 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.518111 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.519637 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.521372 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.521379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.521381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.521716 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.521950 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.521951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.521951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.522091 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.522363 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.522370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.522500 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.522614 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.522780 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260620 18:34:43.523553 17486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:34:43.526664 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.517514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.10.109.124:42029)
0620 18:34:43.517678 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.517681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.517694 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:34:43.517713 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.517714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.517730 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.517792 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.518445 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.519265 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.520134 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.520596 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.520695 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.523504 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.523511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.523512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.523518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.525336 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.525719 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.525721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.525721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.525787 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.526033 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.526034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.526034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.526159 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.526308 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.526380 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.526464 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.124@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
I20260620 18:34:43.530792 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.523795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45306)
0620 18:34:43.523829 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.523840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.523875 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.524238 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.524238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.524389 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.524448 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.524450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.525387 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.525388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.526141 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.526143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.526606 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.526607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.526658 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.527069 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.527073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.528260 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.529770 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.529771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.529771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.529941 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.530151 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.530152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.530153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.530207 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.530472 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.530472 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.530515 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.530602 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.530690 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 18:34:43.530831 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.524040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.10.109.125:45667)
0620 18:34:43.524107 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.524109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.524117 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:43.524147 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.524148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.524161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.524222 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.524542 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.525280 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.526212 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.526530 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.526593 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.528354 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.528355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.528356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.528363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.529684 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.530026 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.530027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.530028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.530081 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.530280 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.530281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.530281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.530388 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.530475 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.530627 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.530702 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:34:43.531533 17486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.124@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:43.537120 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.531776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.0.0.1:53524)
0620 18:34:43.531801 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.531810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.531847 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.531992 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.531993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.532094 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.532119 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.532120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.532721 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.532722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.533190 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.533191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.533509 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.533509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.533543 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.533842 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.533846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.534934 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.536405 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.536405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.536406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.536546 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.536687 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.536688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.536688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.536731 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.536888 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.536888 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.536955 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.537005 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.537068 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260620 18:34:43.537181 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.531800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.10.109.124:42029)
0620 18:34:43.531849 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.531851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.531887 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:34:43.531918 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.531918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.531927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.531965 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.532176 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.532656 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.533258 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.533450 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.533498 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.535006 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.535007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.535008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.535013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.536340 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.536600 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.536600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.536600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.536639 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.536767 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.536767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.536767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.536842 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.536883 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.537020 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.537108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":31}
I20260620 18:34:43.538580 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Bootstrap starting.
I20260620 18:34:43.539628 17486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:43.539968 17486 log.cc:824] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:43.540647 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: No bootstrap required, opened a new log
I20260620 18:34:43.542152 17486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } }
I20260620 18:34:43.542264 17486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:43.542285 17486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d350436b7d4ff48289c2db754a4e72, State: Initialized, Role: FOLLOWER
I20260620 18:34:43.542407 17486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } }
I20260620 18:34:43.542706 17486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:43.542644 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } } }
I20260620 18:34:43.542780 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:34:43.544282 17502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:43.544348 17502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:43.546597 17418 master_runner.cc:428] Detected that this master 93d350436b7d4ff48289c2db754a4e72 is joining an existing cluster
I20260620 18:34:43.546646 17418 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.109.124:42029
I20260620 18:34:43.554139 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.10.109.124:42029)
0620 18:34:43.547834 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.547836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.547955 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:34:43.548025 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.548025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.548034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.548132 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.548538 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.549236 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550335 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550602 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550637 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.551790 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.551790 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.551791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.551796 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.553048 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.553414 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.553415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.553416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.553464 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.553633 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.553634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.553635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.553739 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.553812 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.553959 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.554036 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:34:43.554229 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.0.0.1:53540)
0620 18:34:43.547898 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.547908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.547969 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.548140 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.548141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.548374 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.548470 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.548472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.549324 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.549325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.550240 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.550243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550716 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.550816 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.551265 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.551269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.551682 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.553140 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.553140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.553141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.553331 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.553512 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.553512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.553513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.553563 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.553803 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.553804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.553852 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.553940 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.554087 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":103,"thread_start_us":81,"threads_started":1}
I20260620 18:34:43.578229 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16828
I20260620 18:34:43.578931 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.10.109.125:45667)
0620 18:34:43.548182 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.548184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.548255 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:34:43.548357 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.548358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.548368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.548396 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.548773 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.549218 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550336 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550630 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550667 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.576017 (+ 25350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.576020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.576020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.576029 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.577552 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.577999 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.578001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.578001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.578061 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.578273 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.578273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.578274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.578420 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.578464 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.578661 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.578781 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260620 18:34:43.579300 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45310)
0620 18:34:43.548178 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.548185 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.548282 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.548459 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.548460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.548646 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.548686 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.548687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.549297 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.549298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.550240 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.550243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550733 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.550816 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.575445 (+ 24629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.575450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.575860 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.577661 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.577662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.577663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.577914 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.578139 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.578140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.578140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.578199 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.578510 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.578510 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.578558 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.579008 (+ 450us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.579135 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":67,"thread_start_us":58,"threads_started":1}
I20260620 18:34:43.580510 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.10.109.126:43923)
0620 18:34:43.547884 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.547886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.548131 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:34:43.548167 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.548168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.548178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.548230 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.549529 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550243 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.551322 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.551631 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.551705 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.576028 (+ 24323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.576030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.576030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.576038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.577851 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.578227 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.578228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.578228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.578277 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.579915 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.579916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.579917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.580050 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.580136 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.580327 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.580409 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:43.580539 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.547770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39632)
0620 18:34:43.548055 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.548063 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.548108 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.548282 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.548283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.549369 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.549447 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.549449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.550338 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.550339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.551237 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.551239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.551719 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.551720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.551788 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.575365 (+ 23577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.575371 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.575909 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.577947 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.577948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.577948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.578150 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.579700 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.579701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.579701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.579768 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.580145 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.580146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.580195 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.580267 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.580380 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":55,"threads_started":1}
I20260620 18:34:43.585083 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:33709
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:43923
--webserver_port=40811
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980482, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.126@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:43.591159 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.582943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.0.0.1:53550)
0620 18:34:43.582984 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.582993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.583026 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.583789 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.583790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.583951 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.583996 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.583998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.584907 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.584908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.585768 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.585771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.586281 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.586283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.586352 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.586796 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.586801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.588282 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.590063 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.590064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.590065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.590270 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.590478 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.590479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.590479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.590535 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.590852 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.590853 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.590895 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.590960 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.591059 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 18:34:43.591719 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.583146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.10.109.124:42029)
0620 18:34:43.583667 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.583668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.583676 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:43.583701 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.583702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.583710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.583758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.584086 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.584822 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.585857 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.586161 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.586230 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.588373 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.588375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.588375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.588381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.589973 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.590345 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.590346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.590346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.590399 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.590596 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.590597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.590597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.590726 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.590824 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.591559 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.591633 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502}
I20260620 18:34:43.592396 16766 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.109.124:42029: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "93d350436b7d4ff48289c2db754a4e72"
member_type: NON_VOTER
last_known_addr {
host: "127.10.109.124"
port: 42029
}
attrs {
promote: true
}
}
dest_uuid: "1a673faa5b894849b1cfe935680c6f45"
cas_config_opid_index: 5
I20260620 18:34:43.592645 16766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }
I20260620 18:34:43.592974 17325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.593248 17460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:43.594175 17460 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:34:43.594173 17325 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.109.124:42029
I20260620 18:34:43.594255 17325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.594331 17325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:43.594780 17326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:43.596112 17460 raft_consensus.cc:858] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }
W20260620 18:34:43.596189 17460 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }
I20260620 18:34:43.596791 17460 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.597754 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a673faa5b894849b1cfe935680c6f45. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.597836 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.597901 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 18:34:43.597932 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } }
I20260620 18:34:43.597962 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 18:34:43.597831 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 18:34:43.597982 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.598060 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:43.598273 17459 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260620 18:34:43.598357 17459 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.598929 17418 master.cc:537] Master@127.10.109.124:42029 shutting down...
I20260620 18:34:43.599330 17325 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: attempting to promote NON_VOTER 93d350436b7d4ff48289c2db754a4e72 to VOTER
I20260620 18:34:43.599515 17325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:43.600062 17512 mvcc.cc:205] Tried to move back new op lower bound from 7298992056688517120 to 7298992056602583040. Current Snapshot: MvccSnapshot[applied={T|T < 7298992056612106240}]
I20260620 18:34:43.600189 17514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
W20260620 18:34:43.600288 16753 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 -> Peer 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124:42029): Couldn't send request to peer 93d350436b7d4ff48289c2db754a4e72. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:34:43.600682 17514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:34:43.603041 17418 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 18:34:43.603061 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.594552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45322)
0620 18:34:43.594717 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.594727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.594790 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.595094 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.595095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.595097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:43.595133 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.595134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.600600 (+ 5466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.600601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.601798 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.601802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.602796 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.602797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.602866 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.602885 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.602954 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.602993 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 18:34:43.603093 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.594647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.10.109.125:45667)
0620 18:34:43.594686 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.594688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.594860 (+ 172us) server_negotiation.cc:407] Connection header received
0620 18:34:43.594884 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.594885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.594918 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.594981 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:43.599107 (+ 4126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.599858 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.601958 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.602567 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.602619 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.602686 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.602963 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.603016 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:43.603147 17418 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 18:34:43.603184 17418 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: stopping tablet replica
I20260620 18:34:43.603364 17374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:34:43.603492 17373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:34:43.603665 17326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:34:43.604087 17373 raft_consensus.cc:858] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
W20260620 18:34:43.604147 17373 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:43.604357 17373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.605407 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a673faa5b894849b1cfe935680c6f45. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.605499 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:34:43.605532 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } }
I20260620 18:34:43.605576 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:34:43.605767 17326 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: attempt to promote peer 5faca9ffaeb1442589a58040c0b33b04: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260620 18:34:43.605825 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:43.605906 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:43.606416 17325 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: attempt to promote peer 5faca9ffaeb1442589a58040c0b33b04: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260620 18:34:43.609521 17418 master.cc:552] Master@127.10.109.124:42029 shutdown complete.
I20260620 18:34:43.609568 17418 master_runner.cc:305] Clearing existing system tablet
I20260620 18:34:43.609697 17418 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.609741 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.610495 17418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:43.611173 17523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.611301 17418 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.611346 17418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.611367 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "93d350436b7d4ff48289c2db754a4e72"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "672c076fb24c63f9bd99c55341a62059"
server_key_iv: "6d09d6049171e6796141075ac9e86f0b"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.612102 17418 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:43.613457 17418 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:34:43.613525 17418 log.cc:1196] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:34:43.614058 17418 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Deleting consensus metadata
I20260620 18:34:43.614388 17418 master_runner.cc:315] Copying system tablet from 127.10.109.126:43923
I20260620 18:34:43.615046 17418 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: tablet copy: Beginning tablet copy session from remote peer at address 127.10.109.126:43923
I20260620 18:34:43.625905 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.618944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.10.109.126:43923)
0620 18:34:43.618993 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.618995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.619098 (+ 103us) server_negotiation.cc:407] Connection header received
0620 18:34:43.619155 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.619156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.619166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.619252 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.619606 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.620368 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.621368 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.621655 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.621736 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.622895 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.622897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.622898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.622906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.624612 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.625061 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.625062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.625063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.625124 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.625363 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.625365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.625365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.625495 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.625563 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.625729 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.625805 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:34:43.625910 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.618896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:39648)
0620 18:34:43.619022 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.619033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.619096 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.619266 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.619267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.619464 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.619519 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.619521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.620462 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.620463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.621273 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.621275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.621741 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.621742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.621801 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.622368 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.622373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.622804 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.624715 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.624716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.624716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.624954 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.625228 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.625229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.625229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.625296 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.625566 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.625567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.625616 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.625689 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.625799 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260620 18:34:43.626561 16786 tablet_copy_service.cc:140] P 1a673faa5b894849b1cfe935680c6f45: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 93d350436b7d4ff48289c2db754a4e72 ({username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:39648)
I20260620 18:34:43.626618 16786 tablet_copy_service.cc:161] P 1a673faa5b894849b1cfe935680c6f45: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 93d350436b7d4ff48289c2db754a4e72 at {username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:39648: session id = 93d350436b7d4ff48289c2db754a4e72-00000000000000000000000000000000
I20260620 18:34:43.627315 16786 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:34:43.627825 17418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:34:43.629254 17418 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: tablet copy: Starting download of 0 data blocks...
I20260620 18:34:43.629408 17418 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: tablet copy: Starting download of 1 WAL segments...
I20260620 18:34:43.630121 17418 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:34:43.631052 16786 tablet_copy_service.cc:342] P 1a673faa5b894849b1cfe935680c6f45: Request end of tablet copy session 93d350436b7d4ff48289c2db754a4e72-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:39648
I20260620 18:34:43.631107 16786 tablet_copy_service.cc:433] P 1a673faa5b894849b1cfe935680c6f45: ending tablet copy session 93d350436b7d4ff48289c2db754a4e72-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 93d350436b7d4ff48289c2db754a4e72
I20260620 18:34:43.633414 17418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.634946 17530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.634980 17533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.638957 17418 server_base.cc:1034] running on GCE node
W20260620 18:34:43.639636 17531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.639825 17418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.639987 17418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.641110 17418 hybrid_clock.cc:648] HybridClock initialized: now 1781980483641099 us; error 27 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.642812 17418 init.cc:377] Logged in from keytab as oryx/127.10.109.124@KRBTEST.COM (short username oryx)
I20260620 18:34:43.643076 17418 webserver.cc:460] Webserver started at http://127.10.109.124:37583/ using document root <none> and password file <none>
I20260620 18:34:43.643198 17418 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.643239 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.644014 17418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:43.644618 17540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.644743 17418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:43.644781 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "93d350436b7d4ff48289c2db754a4e72"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "672c076fb24c63f9bd99c55341a62059"
server_key_iv: "6d09d6049171e6796141075ac9e86f0b"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.644838 17418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.657060 17418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.658874 17418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.661871 17418 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.124:42029
I20260620 18:34:43.661902 17592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.124:42029 every 8 connection(s)
I20260620 18:34:43.662194 17418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
I20260620 18:34:43.663102 17593 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:34:43.663690 17593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Bootstrap starting.
I20260620 18:34:43.666188 17593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260620 18:34:43.666414 17593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Bootstrap complete.
I20260620 18:34:43.666615 17593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } }
I20260620 18:34:43.666808 17593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d350436b7d4ff48289c2db754a4e72, State: Initialized, Role: FOLLOWER
I20260620 18:34:43.666857 17593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:43.667027 17593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:43.668322 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:43.668481 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:43.669243 17607 catalog_manager.cc:1259] Loaded cluster ID: a335d32e3b494575aaee5e639a2be3e6
I20260620 18:34:43.669322 17607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: loading cluster ID for follower catalog manager: success
I20260620 18:34:43.670542 17607 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: acquiring CA information for follower catalog manager: success
I20260620 18:34:43.670869 17607 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:34:43.690366 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.690549 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.690572 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.691749 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.691789 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.691814 17511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:43.693266 17511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:33709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=40811
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:43.693511 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.693747 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.699536 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.699548 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.699836 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.699832 17511 server_base.cc:1034] running on GCE node
I20260620 18:34:43.700183 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.700419 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.701588 17511 hybrid_clock.cc:648] HybridClock initialized: now 1781980483701570 us; error 32 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.704089 17511 init.cc:377] Logged in from keytab as oryx/127.10.109.65@KRBTEST.COM (short username oryx)
I20260620 18:34:43.704576 17511 webserver.cc:460] Webserver started at http://127.10.109.65:40811/ using document root <none> and password file <none>
I20260620 18:34:43.704818 17511 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.704903 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.706233 17511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.706966 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.707152 17511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.707247 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "f391411f1fe24c91864a8e92daf750fd"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "e1f107748acb491cd724f005ccdfa4b6"
server_key_iv: "343d286c9bc04402a88730949cdc0487"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.707621 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.725297 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.727766 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:43.727954 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.728233 17511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:43.728538 17511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:43.728591 17511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.728626 17511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:43.728662 17511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.734308 17511 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:33709
I20260620 18:34:43.734349 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:33709 every 8 connection(s)
I20260620 18:34:43.734738 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:43.740759 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17511
I20260620 18:34:43.740921 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16962
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:43.747141 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:57217 (local address 127.10.109.126:43923)
0620 18:34:43.735998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.735999 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.736008 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:43.736968 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.736969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.736983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.737013 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.738466 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.739269 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.740338 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.740613 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.740678 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.743668 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.743670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.743670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.743677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.745340 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.746057 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.746059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.746059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.746119 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.746381 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.746382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.746382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.746513 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.746592 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.746951 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.747031 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:43.747527 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:37641
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:43923
--webserver_port=43803
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:34:43.747961 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:57217)
0620 18:34:43.735959 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.735967 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.736853 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.737156 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.737160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.737551 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.737750 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.737755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.739367 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.739368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.740214 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.740215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.740756 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.740758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.740878 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.741674 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.741679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.743578 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.745473 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.745480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.745482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.745942 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.746193 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.746194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.746194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.746309 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.746583 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.746589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.746778 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.747523 (+ 745us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.747701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":6,"thread_start_us":53,"threads_started":1}
I20260620 18:34:43.748440 17737 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:43.748492 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:34305 (local address 127.10.109.125:45667)
0620 18:34:43.735604 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.735606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.735847 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:34:43.736983 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.736984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.736996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.737032 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.737849 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.738335 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.739695 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.740009 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.744673 (+ 4664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.744738 (+ 65us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.744739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.744739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.744744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.746542 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.747429 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.747431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.747431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.747487 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.747913 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.747926 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.747927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.748077 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.748136 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.748314 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.748384 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:43.748530 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.10.109.65:34305)
0620 18:34:43.735740 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.735762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.736844 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.737162 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.737163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.737554 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.737749 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.737753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.738444 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.738445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.739607 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.739610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.740092 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.740093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.740231 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.740800 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.740812 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.743266 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.746834 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.746835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.746835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.747341 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.747780 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.747780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.747781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.747842 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.748174 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.748174 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.748213 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.748275 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.748375 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260620 18:34:43.748559 17737 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.748752 17737 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:43.749128 17738 heartbeater.cc:347] Connected to a master server at 127.10.109.125:45667
I20260620 18:34:43.749177 17738 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.749238 17738 heartbeater.cc:511] Master 127.10.109.125:45667 requested a full tablet report, sending...
I20260620 18:34:43.749229 16766 ts_manager.cc:194] Re-registered known tserver with Master: f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709)
I20260620 18:34:43.749625 16766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:57217
I20260620 18:34:43.749650 17364 ts_manager.cc:194] Registered new tserver with Master: f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709)
I20260620 18:34:43.752003 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:50379 (local address 127.10.109.124:42029)
0620 18:34:43.735820 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.735822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.735832 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:43.736981 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.736983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.736996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.737069 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.738809 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.739729 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.740789 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.741185 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.741248 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.746644 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.746646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.746646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.746654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.750918 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.751291 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.751292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.751292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.751342 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.751564 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.751564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.751565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.751663 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.751712 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.751873 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.751940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":50,"threads_started":1}
I20260620 18:34:43.752261 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.735556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.10.109.65:50379)
0620 18:34:43.735720 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.735762 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.736842 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.737155 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.737160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.737552 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.737750 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.737755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.739899 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.739900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.740687 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.740690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.741301 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.741302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.741374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.741821 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.741826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.745769 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.750993 (+ 5224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.750994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.750994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.751217 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.751406 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.751407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.751407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.751503 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.751726 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.751727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.751771 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.751849 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.751968 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260620 18:34:43.752750 17736 heartbeater.cc:347] Connected to a master server at 127.10.109.124:42029
I20260620 18:34:43.752810 17736 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.752882 17736 heartbeater.cc:511] Master 127.10.109.124:42029 requested a full tablet report, sending...
I20260620 18:34:43.753201 17557 ts_manager.cc:194] Registered new tserver with Master: f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709)
W20260620 18:34:43.830483 17744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.830703 17744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.830742 17744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.831923 17744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.831985 17744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.832031 17744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:43.833493 17744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:37641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=43803
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:43.833734 17744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.834028 17744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.839444 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.839438 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.839478 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.840040 17744 server_base.cc:1034] running on GCE node
I20260620 18:34:43.840464 17744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.840703 17744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.841890 17744 hybrid_clock.cc:648] HybridClock initialized: now 1781980483841877 us; error 31 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.844976 17744 init.cc:377] Logged in from keytab as oryx/127.10.109.66@KRBTEST.COM (short username oryx)
I20260620 18:34:43.845404 17744 webserver.cc:460] Webserver started at http://127.10.109.66:43803/ using document root <none> and password file <none>
I20260620 18:34:43.845582 17744 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.845633 17744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.847072 17744 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.847645 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.847832 17744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:43.847925 17744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "4560515d11024407b702f9acb1e19eff"
format_stamp: "Formatted at 2026-06-20 18:34:42 on dist-test-slave-3jql"
server_key: "ab35b4732fde58c6d0707fa6f53c4c08"
server_key_iv: "e4ee67c78846cd281975986542a7fbc1"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.848259 17744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.866717 17744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:43.869377 17744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:43.869540 17744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:43.869800 17744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:43.870131 17744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:43.870163 17744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.870208 17744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:43.870234 17744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.875907 17744 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37641
I20260620 18:34:43.876006 17872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37641 every 8 connection(s)
I20260620 18:34:43.876340 17744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:43.884477 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17744
I20260620 18:34:43.884634 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17096
I20260620 18:34:43.887279 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:39351 (local address 127.10.109.125:45667)
0620 18:34:43.877598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.877600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.877733 (+ 133us) server_negotiation.cc:407] Connection header received
0620 18:34:43.878084 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.878085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.878098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.878132 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.878867 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.879389 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880483 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.880687 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880760 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.883356 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.883357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.883357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.883364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.885739 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.886168 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.886170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.886170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.886229 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.886497 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.886498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.886498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.886617 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.886691 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.887042 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.887132 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:34:43.887434 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.10.109.66:39351)
0620 18:34:43.877669 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.877677 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.877983 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.878205 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.878214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.878569 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.878773 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.878779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.879509 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.879510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.880366 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.880368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.880778 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.880893 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.881381 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.881388 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.883220 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.885839 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.885847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.885849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.886076 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.886296 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.886297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.886297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.886413 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.886695 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.886702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.886890 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.887002 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.887183 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":32,"thread_start_us":62,"threads_started":1}
I20260620 18:34:43.888304 17875 heartbeater.cc:347] Connected to a master server at 127.10.109.125:45667
I20260620 18:34:43.888456 17875 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.888680 17875 heartbeater.cc:511] Master 127.10.109.125:45667 requested a full tablet report, sending...
I20260620 18:34:43.889223 17364 ts_manager.cc:194] Registered new tserver with Master: 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:43.890146 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:41949 (local address 127.10.109.126:43923)
0620 18:34:43.877312 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.877314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.877384 (+ 70us) server_negotiation.cc:407] Connection header received
0620 18:34:43.878072 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.878073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.878088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.878137 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.878946 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.879415 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880588 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.880892 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880943 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.884213 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.884215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.884215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.884224 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.888567 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889241 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.889243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.889243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.889304 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889574 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.889575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.889576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.889719 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.889763 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.889970 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.890053 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260620 18:34:43.890228 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:34819
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:43923
--webserver_port=41905
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:34:43.890385 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:41529 (local address 127.10.109.124:42029)
0620 18:34:43.877108 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:43.877110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:43.877632 (+ 522us) server_negotiation.cc:407] Connection header received
0620 18:34:43.878084 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:43.878085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:43.878100 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:43.878132 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:43.879504 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.880250 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.881239 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.881564 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.881663 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.884503 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:43.884505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:43.884505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:43.884512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:43.889212 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889595 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.889596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.889596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.889653 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889861 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:43.889862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:43.889862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:43.889992 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:43.890054 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:43.890231 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:34:43.890321 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:34:43.890182 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:41949)
0620 18:34:43.877258 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.877300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.877960 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.878204 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.878208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.878525 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.878784 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.878787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.879519 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.879520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.880499 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.880500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.881016 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.881016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.881081 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.881515 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.881519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.884122 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.888917 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.888917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.888918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.889148 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.889422 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.889423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.889487 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.889809 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.889809 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.889850 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.889938 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.890053 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":55,"thread_start_us":111,"threads_started":1}
I20260620 18:34:43.891052 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:43.877068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.10.109.66:41529)
0620 18:34:43.877568 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:43.877579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:43.877983 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:43.878205 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:43.878214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:43.878673 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:43.878773 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.878779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.880345 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.880346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.881155 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:43.881157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:43.881666 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:43.881667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:43.881732 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:43.882115 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:43.882120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:43.884435 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:43.889296 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.889298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.889500 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.889729 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:43.889730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:43.889731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:43.889790 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:43.890080 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:43.890081 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:43.890119 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:34:43.890563 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 18:34:43.890665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260620 18:34:43.891463 17873 heartbeater.cc:347] Connected to a master server at 127.10.109.124:42029
I20260620 18:34:43.891527 17873 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.891602 17873 heartbeater.cc:511] Master 127.10.109.124:42029 requested a full tablet report, sending...
I20260620 18:34:43.891614 17874 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:43.891644 17874 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:43.891706 17874 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:43.891883 17557 ts_manager.cc:194] Registered new tserver with Master: 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:43.891949 16766 ts_manager.cc:194] Re-registered known tserver with Master: 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:43.892474 16766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:41949
W20260620 18:34:43.972810 17880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:43.973001 17880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:43.973024 17880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:43.974088 17880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:43.974124 17880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:43.974149 17880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:43.975664 17880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:34819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=41905
--tserver_master_addrs=127.10.109.126:43923,127.10.109.125:45667,127.10.109.124:42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:43.975854 17880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:43.976018 17880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:43.981038 17888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.981179 17880 server_base.cc:1034] running on GCE node
W20260620 18:34:43.981070 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:43.981062 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:43.981699 17880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:43.981909 17880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:43.983057 17880 hybrid_clock.cc:648] HybridClock initialized: now 1781980483983043 us; error 29 us; skew 500 ppm
Jun 20 18:34:43 dist-test-slave-3jql krb5kdc[16721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:43.985707 17880 init.cc:377] Logged in from keytab as oryx/127.10.109.67@KRBTEST.COM (short username oryx)
I20260620 18:34:43.986207 17880 webserver.cc:460] Webserver started at http://127.10.109.67:41905/ using document root <none> and password file <none>
I20260620 18:34:43.986451 17880 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:43.986536 17880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:43.988024 17880 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.988685 17895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:43.988943 17880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:43.989046 17880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
format_stamp: "Formatted at 2026-06-20 18:34:43 on dist-test-slave-3jql"
server_key: "9614cbfa16733c21eddbaad15e65a682"
server_key_iv: "3065a922f15ac1ef006b56dbbf2480d2"
server_key_version: "encryptionkey@0"
I20260620 18:34:43.989413 17880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:43.998582 17880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:44.001046 17880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:44.001204 17880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:44.001438 17880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:44.001732 17880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:44.001786 17880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:44.001953 17880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:44.002089 17880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:44.008932 17880 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:34819
I20260620 18:34:44.008967 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:34819 every 8 connection(s)
I20260620 18:34:44.009311 17880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:44.015899 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 17880
Jun 20 18:34:44 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.125@KRBTEST.COM
Jun 20 18:34:44 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
Jun 20 18:34:44 dist-test-slave-3jql krb5kdc[16721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980483, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.124@KRBTEST.COM
I20260620 18:34:44.025003 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:57713 (local address 127.10.109.125:45667)
0620 18:34:44.010110 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.010113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.010373 (+ 260us) server_negotiation.cc:407] Connection header received
0620 18:34:44.011748 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.011750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.011761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.011793 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.012801 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.013566 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015739 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.015973 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.016058 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.019997 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.019999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.019999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.020005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.022136 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.023756 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.023758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.023758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.023831 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.024172 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.024173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.024173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.024312 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.024376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.024812 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.024899 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:44.025172 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:56201)
0620 18:34:44.010566 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.010576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.011607 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.012370 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.012371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.012531 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:44.012692 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.012696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.013420 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.013421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.014711 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.014713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.015679 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.016734 (+ 1054us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.017100 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:44.017104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:44.020320 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:44.023067 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.023068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.023068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.023782 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.024004 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.024005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.024005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.024075 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.024383 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:44.024391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:44.024578 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.024690 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.024905 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":7,"thread_start_us":61,"threads_started":1}
I20260620 18:34:44.025640 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.10.109.67:57713)
0620 18:34:44.010254 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.010294 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.011587 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.011925 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.011930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.012392 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:44.012704 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.012705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.014838 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.014839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.015639 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.015642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.016114 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.016115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.016179 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.016614 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:44.016623 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:44.019901 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:44.022571 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.022580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.022583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.022934 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.023904 (+ 970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.023905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.023905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.024053 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.024405 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:44.024407 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:44.024578 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.024691 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.025540 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":720,"thread_start_us":97,"threads_started":1}
I20260620 18:34:44.025991 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:56201 (local address 127.10.109.126:43923)
0620 18:34:44.010548 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.010550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.010618 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:34:44.012286 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.012287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.012300 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.012361 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.012795 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.013313 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015278 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.015601 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015672 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.020406 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.020407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.020407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.020414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.022983 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.023870 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.023872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.023872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.023932 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.024149 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.024150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.024150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.024290 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.024352 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.025810 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.025900 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:34:44.026057 18011 heartbeater.cc:347] Connected to a master server at 127.10.109.125:45667
I20260620 18:34:44.026096 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.017573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.10.109.126:43923)
0620 18:34:44.017716 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.017718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.018497 (+ 779us) server_negotiation.cc:407] Connection header received
0620 18:34:44.018565 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.018566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.018584 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.018666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.019290 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.020040 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.021187 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.021515 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.021561 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.023052 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.023053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.023054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.023060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.024792 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.025240 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.025241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.025242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.025305 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.025519 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.025520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.025520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.025645 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.025726 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.025945 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.026015 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260620 18:34:44.026145 18011 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:44.026342 18011 heartbeater.cc:511] Master 127.10.109.125:45667 requested a full tablet report, sending...
I20260620 18:34:44.026855 18010 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43923
I20260620 18:34:44.026898 18010 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:44.026958 18010 heartbeater.cc:511] Master 127.10.109.126:43923 requested a full tablet report, sending...
I20260620 18:34:44.027209 17364 ts_manager.cc:194] Registered new tserver with Master: f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:44.027372 16765 ts_manager.cc:194] Re-registered known tserver with Master: f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:44.027904 16765 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:56201
I20260620 18:34:44.028002 17519 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: attempt to promote peer 5faca9ffaeb1442589a58040c0b33b04: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260620 18:34:44.029112 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.018256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.10.109.125:45667)
0620 18:34:44.018361 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.018362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.020053 (+ 1691us) server_negotiation.cc:407] Connection header received
0620 18:34:44.020074 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.020075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.020085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.020132 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.020877 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.021671 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.022769 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.023103 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.023160 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.025224 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.025225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.025225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.025232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.027822 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.028266 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.028267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.028268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.028333 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.028518 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.028519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.028519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.028635 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.028719 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.028915 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.029006 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260620 18:34:44.029122 16765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:44.031100 16765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:44.031629 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.017863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.10.109.124:42029)
0620 18:34:44.018609 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.018611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.018619 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:44.018638 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.018639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.018649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.018707 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.019466 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.020187 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.026972 (+ 6785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.027246 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.027312 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.028712 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.028713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.028714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.028721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.030347 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.030701 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.030702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.030702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.030782 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.031097 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.031098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.031098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.031242 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.031339 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.031445 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.031534 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":58,"threads_started":1}
I20260620 18:34:44.032044 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:54051 (local address 127.10.109.124:42029)
0620 18:34:44.010363 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.010365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.010374 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:44.011721 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.011723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.011737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.011770 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.012801 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.013556 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.014839 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.015143 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015220 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.021180 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.021181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.021182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.021189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.023676 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.031260 (+ 7584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.031261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.031262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.031309 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.031518 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.031519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.031519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.031627 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.031676 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.031868 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.031953 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:34:44.032593 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.010032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.10.109.67:54051)
0620 18:34:44.010254 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.010296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.011587 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.011920 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.011932 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.012392 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:44.012693 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.012699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.013633 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.013634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.014711 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.014713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.015227 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.015228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.015335 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.016639 (+ 1304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:44.016644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:44.021051 (+ 4407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:44.030922 (+ 9871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.030922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.030923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.031171 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.031375 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:44.031376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:44.031376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:44.031445 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:44.031712 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:44.031713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:44.031754 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.032154 (+ 400us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.032280 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":81,"threads_started":1}
I20260620 18:34:44.033252 18009 heartbeater.cc:347] Connected to a master server at 127.10.109.124:42029
I20260620 18:34:44.033308 18009 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:44.033373 18009 heartbeater.cc:511] Master 127.10.109.124:42029 requested a full tablet report, sending...
I20260620 18:34:44.033428 17326 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: attempt to promote peer 5faca9ffaeb1442589a58040c0b33b04: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260620 18:34:44.033659 17557 ts_manager.cc:194] Registered new tserver with Master: f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:44.056605 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.053938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.0.0.1:53564)
0620 18:34:44.054024 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.054037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.054110 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.054291 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.054292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.054294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.054349 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.054352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.055095 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.055096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.055797 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.055799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.056367 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.056368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.056431 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.056434 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.056493 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.056544 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:34:44.056604 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.054066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.10.109.124:42029)
0620 18:34:44.054133 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.054136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.054147 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:44.054172 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.054173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.054201 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.054286 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.054418 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.055009 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.055896 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.056278 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.056367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.056435 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.056499 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.056534 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:34:44.057016 17567 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.8] Dedup: 1.7->[1.8-1.8]
I20260620 18:34:44.057446 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a673faa5b894849b1cfe935680c6f45. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: NON_VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.057572 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:44.057979 17519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.058135 17567 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.059233 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.059340 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.059527 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.059546 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.059608 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.059616 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:44.059592 17373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.060673 17521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: NON_VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: true } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.060777 17521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:34:44.060926 18032 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: attempting to promote NON_VOTER 5faca9ffaeb1442589a58040c0b33b04 to VOTER
I20260620 18:34:44.061050 18032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:44.061427 17567 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 18:34:44.061431 17373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 LEARNER]: Refusing update from remote peer 1a673faa5b894849b1cfe935680c6f45: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 18:34:44.061612 17326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 18:34:44.061703 17519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 18:34:44.062179 17326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.062377 17373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.063102 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.063205 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.063098 18032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.063337 18032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.063156 17325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.063835 17325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:44.063930 17521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.064016 17521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:44.064178 17567 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.065097 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a673faa5b894849b1cfe935680c6f45" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:44.065173 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:44.067337 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.063371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34819 (local address 127.0.0.1:33432)
0620 18:34:44.063498 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.063509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.063558 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.064212 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.064212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.064213 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.064263 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.064265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.065279 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.065281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.066340 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.066343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.067086 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.067087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.067160 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.067165 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.067223 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.067261 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":37,"thread_start_us":80,"threads_started":1}
I20260620 18:34:44.067345 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.063813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.10.109.67:34819)
0620 18:34:44.063968 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.063973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.063989 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:34:44.064050 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.064052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.064070 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.064338 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.064397 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.065158 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.066403 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.067003 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.067086 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.067158 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.067237 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.067274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260620 18:34:44.067942 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.063563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37641 (local address 127.0.0.1:38088)
0620 18:34:44.063666 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.063674 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.063712 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.064209 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.064210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.064212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.064263 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.064266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.065462 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.065464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.066719 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.066722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.067720 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.067721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.067778 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.067782 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.067812 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.067846 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260620 18:34:44.068065 17943 tablet_service.cc:1467] Processing CreateTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b (DEFAULT_TABLE table=test-table [id=15cc37af19844dcd8edf8900681c6484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:44.068367 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.063660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.10.109.66:37641)
0620 18:34:44.064007 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.064013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.064024 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:44.064065 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.064067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.064084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.064190 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.064326 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.065365 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.066847 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.067596 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.068106 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.068182 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.068207 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.068244 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":77,"threads_started":1}
I20260620 18:34:44.068440 17943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d9664eccb4ea1afea8f347de6fd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:44.069047 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.062664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33709 (local address 127.0.0.1:40250)
0620 18:34:44.062806 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.062817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.062856 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.065538 (+ 2682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.065539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.065541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.065571 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.065573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.066815 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.066816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.067876 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.067878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.068852 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.068853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.068915 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.068918 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.068949 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.068988 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1}
I20260620 18:34:44.068975 17807 tablet_service.cc:1467] Processing CreateTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b (DEFAULT_TABLE table=test-table [id=15cc37af19844dcd8edf8900681c6484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:44.069288 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d9664eccb4ea1afea8f347de6fd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:44.069499 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.062808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.10.109.65:33709)
0620 18:34:44.065380 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.065384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.065394 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:44.065432 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.065434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.065450 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.065529 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.065635 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.066674 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.067987 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.068758 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.069210 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.069326 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.069355 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.069423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2527,"thread_start_us":97,"threads_started":1}
I20260620 18:34:44.070073 17670 tablet_service.cc:1467] Processing CreateTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b (DEFAULT_TABLE table=test-table [id=15cc37af19844dcd8edf8900681c6484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:44.070400 17670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d9664eccb4ea1afea8f347de6fd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:44.071743 18039 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Bootstrap starting.
I20260620 18:34:44.072331 18039 tablet_bootstrap.cc:654] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:44.072641 18039 log.cc:824] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:44.073314 18041 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Bootstrap starting.
I20260620 18:34:44.073381 18039 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: No bootstrap required, opened a new log
I20260620 18:34:44.073443 18039 ts_tablet_manager.cc:1397] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:44.074061 18041 tablet_bootstrap.cc:654] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:44.074375 18041 log.cc:824] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:44.074914 18042 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Bootstrap starting.
I20260620 18:34:44.075142 18041 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: No bootstrap required, opened a new log
I20260620 18:34:44.075217 18041 ts_tablet_manager.cc:1397] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:44.075479 18042 tablet_bootstrap.cc:654] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:44.075757 18042 log.cc:824] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:44.076220 18039 raft_consensus.cc:348] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.076377 18039 raft_consensus.cc:374] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:44.076412 18039 raft_consensus.cc:729] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86f7bc8eafc435bb9d374abf90ad5bd, State: Initialized, Role: FOLLOWER
I20260620 18:34:44.076543 18039 consensus_queue.cc:260] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.076794 18039 ts_tablet_manager.cc:1428] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:44.076820 18010 heartbeater.cc:503] Master 127.10.109.126:43923 was elected leader, sending a full tablet report...
I20260620 18:34:44.078143 18041 raft_consensus.cc:348] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.078253 18041 raft_consensus.cc:374] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:44.078274 18041 raft_consensus.cc:729] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4560515d11024407b702f9acb1e19eff, State: Initialized, Role: FOLLOWER
I20260620 18:34:44.078373 18041 consensus_queue.cc:260] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.078639 18041 ts_tablet_manager.cc:1428] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 18:34:44.078673 17874 heartbeater.cc:503] Master 127.10.109.126:43923 was elected leader, sending a full tablet report...
I20260620 18:34:44.079270 18042 tablet_bootstrap.cc:492] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: No bootstrap required, opened a new log
I20260620 18:34:44.079336 18042 ts_tablet_manager.cc:1397] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 18:34:44.081130 18042 raft_consensus.cc:348] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.081264 18042 raft_consensus.cc:374] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:44.081308 18042 raft_consensus.cc:729] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f391411f1fe24c91864a8e92daf750fd, State: Initialized, Role: FOLLOWER
I20260620 18:34:44.081422 18042 consensus_queue.cc:260] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.081663 18042 ts_tablet_manager.cc:1428] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:34:44.081722 17737 heartbeater.cc:503] Master 127.10.109.126:43923 was elected leader, sending a full tablet report...
I20260620 18:34:44.084946 18047 raft_consensus.cc:482] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:44.085016 18047 raft_consensus.cc:504] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.085398 18047 leader_election.cc:288] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819), 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:44.088770 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.085988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:58309 (local address 127.10.109.66:37641)
0620 18:34:44.086054 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.086056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.086064 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:44.086084 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.086085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.086095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.086150 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.086521 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.087091 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088021 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.088417 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088476 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.088482 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.088665 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.088708 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:34:44.088878 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.085678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34819 (local address 127.10.109.65:39393)
0620 18:34:44.085733 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.085745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.085826 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.085968 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.085969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.085970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.086054 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.086056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.087109 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.087111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.087934 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.087936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.088567 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.088662 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.088668 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.088730 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.088778 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":26}
I20260620 18:34:44.088886 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.085763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:39393 (local address 127.10.109.67:34819)
0620 18:34:44.085849 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.085851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.085862 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:44.085893 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.085894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.085903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.085967 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:44.086249 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.087016 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088061 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.088480 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088561 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.088583 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.088754 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.088805 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:34:44.088963 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.085795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37641 (local address 127.10.109.65:58309)
0620 18:34:44.085843 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:44.085855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:44.085905 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:44.086250 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:44.086252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:44.086256 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:44.086362 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.086364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.087192 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.087193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.087927 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:44.087929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.088503 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.088504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:44.088583 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.088589 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:34:44.088644 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:34:44.088694 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 18:34:44.089308 17963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d9664eccb4ea1afea8f347de6fd4b" candidate_uuid: "f391411f1fe24c91864a8e92daf750fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" is_pre_election: true
I20260620 18:34:44.089473 17963 raft_consensus.cc:2454] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f391411f1fe24c91864a8e92daf750fd in term 0.
I20260620 18:34:44.089501 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d9664eccb4ea1afea8f347de6fd4b" candidate_uuid: "f391411f1fe24c91864a8e92daf750fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4560515d11024407b702f9acb1e19eff" is_pre_election: true
I20260620 18:34:44.089608 17827 raft_consensus.cc:2454] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f391411f1fe24c91864a8e92daf750fd in term 0.
I20260620 18:34:44.089699 17625 leader_election.cc:302] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f391411f1fe24c91864a8e92daf750fd, f86f7bc8eafc435bb9d374abf90ad5bd; no voters:
I20260620 18:34:44.089835 18047 raft_consensus.cc:2781] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:44.089897 18047 raft_consensus.cc:482] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:44.089921 18047 raft_consensus.cc:3037] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:44.090613 18047 raft_consensus.cc:504] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.090737 18047 leader_election.cc:288] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [CANDIDATE]: Term 1 election: Requested vote from peers f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819), 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641)
I20260620 18:34:44.090899 17963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d9664eccb4ea1afea8f347de6fd4b" candidate_uuid: "f391411f1fe24c91864a8e92daf750fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
I20260620 18:34:44.090929 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d9664eccb4ea1afea8f347de6fd4b" candidate_uuid: "f391411f1fe24c91864a8e92daf750fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4560515d11024407b702f9acb1e19eff"
I20260620 18:34:44.090975 17963 raft_consensus.cc:3037] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:44.091002 17827 raft_consensus.cc:3037] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:44.091785 17963 raft_consensus.cc:2454] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f391411f1fe24c91864a8e92daf750fd in term 1.
I20260620 18:34:44.091794 17827 raft_consensus.cc:2454] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f391411f1fe24c91864a8e92daf750fd in term 1.
I20260620 18:34:44.091986 17625 leader_election.cc:302] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4560515d11024407b702f9acb1e19eff, f391411f1fe24c91864a8e92daf750fd; no voters:
I20260620 18:34:44.092092 18047 raft_consensus.cc:2781] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:44.092214 18047 raft_consensus.cc:686] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 1 LEADER]: Becoming Leader. State: Replica: f391411f1fe24c91864a8e92daf750fd, State: Running, Role: LEADER
I20260620 18:34:44.092298 18047 consensus_queue.cc:237] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } }
I20260620 18:34:44.092993 16765 catalog_manager.cc:5495] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd reported cstate change: term changed from 0 to 1, leader changed from <none> to f391411f1fe24c91864a8e92daf750fd (127.10.109.65). New cstate: current_term: 1 leader_uuid: "f391411f1fe24c91864a8e92daf750fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:44.102533 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.100240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.10.109.65:33709)
0620 18:34:44.100300 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.100302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.100312 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:44.100390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.100391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.100405 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.100463 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:44.100566 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.101020 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.101757 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.101961 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.102019 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.102204 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:44.102294 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:44.102369 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.102413 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.102432 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:44.104291 17963 raft_consensus.cc:1261] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Refusing update from remote peer f391411f1fe24c91864a8e92daf750fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:44.104315 17827 raft_consensus.cc:1261] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Refusing update from remote peer f391411f1fe24c91864a8e92daf750fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:44.104571 18047 consensus_queue.cc:1035] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:44.104697 18048 consensus_queue.cc:1035] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:44.107684 18053 mvcc.cc:205] Tried to move back new op lower bound from 7298992062888513536 to 7298992062842544128. Current Snapshot: MvccSnapshot[applied={T|T < 7298992062888513536}]
I20260620 18:34:44.107735 18051 mvcc.cc:205] Tried to move back new op lower bound from 7298992062888513536 to 7298992062842544128. Current Snapshot: MvccSnapshot[applied={T|T < 7298992062888513536}]
I20260620 18:34:44.108253 18052 mvcc.cc:205] Tried to move back new op lower bound from 7298992062888513536 to 7298992062842544128. Current Snapshot: MvccSnapshot[applied={T|T < 7298992062888513536}]
I20260620 18:34:44.110231 16766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:44.110318 16766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:44.111553 16766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a673faa5b894849b1cfe935680c6f45: Sending DeleteTablet for 3 replicas of tablet a34d9664eccb4ea1afea8f347de6fd4b
I20260620 18:34:44.111850 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:44 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:40250
I20260620 18:34:44.111881 17943 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:44 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:33432
I20260620 18:34:44.112131 17807 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d9664eccb4ea1afea8f347de6fd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:44 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:38088
I20260620 18:34:44.112464 18062 tablet_replica.cc:330] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: stopping tablet replica
I20260620 18:34:44.112560 18062 raft_consensus.cc:2229] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:44.112640 18062 raft_consensus.cc:2258] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:44.112869 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 16734
I20260620 18:34:44.113018 18063 tablet_replica.cc:330] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: stopping tablet replica
I20260620 18:34:44.113123 18061 tablet_replica.cc:330] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: stopping tablet replica
I20260620 18:34:44.113270 18061 raft_consensus.cc:2229] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:44.113382 18063 raft_consensus.cc:2229] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:44.113526 18061 raft_consensus.cc:2258] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:44.113649 18063 raft_consensus.cc:2258] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:44.114001 18062 ts_tablet_manager.cc:1905] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:44.114213 18061 ts_tablet_manager.cc:1905] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:44.115635 18063 ts_tablet_manager.cc:1905] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:44.117790 18062 ts_tablet_manager.cc:1918] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:44.117863 18062 log.cc:1196] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/a34d9664eccb4ea1afea8f347de6fd4b
I20260620 18:34:44.118124 18062 ts_tablet_manager.cc:1939] T a34d9664eccb4ea1afea8f347de6fd4b P f391411f1fe24c91864a8e92daf750fd: Deleting consensus metadata
I20260620 18:34:44.119206 18063 ts_tablet_manager.cc:1918] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:44.119277 18063 log.cc:1196] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/a34d9664eccb4ea1afea8f347de6fd4b
I20260620 18:34:44.119527 18063 ts_tablet_manager.cc:1939] T a34d9664eccb4ea1afea8f347de6fd4b P 4560515d11024407b702f9acb1e19eff: Deleting consensus metadata
W20260620 18:34:44.119871 17763 connection.cc:441] server connection from 127.0.0.1:38088 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38088 (request call id 1) could send its response
W20260620 18:34:44.120082 17623 connection.cc:535] server connection from 127.0.0.1:40250 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:34:44.122563 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.112772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.10.109.124:42029)
0620 18:34:44.112814 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.112816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.114043 (+ 1227us) server_negotiation.cc:407] Connection header received
0620 18:34:44.114194 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.114195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.114206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.114235 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.115049 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.115783 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.116936 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.117235 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.117280 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.119057 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.119058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.119058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.119065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.120891 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.121375 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.121377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.121377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.121445 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.121636 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.121637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.121637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.121805 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.121914 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.122357 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.122447 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:44.126859 18061 ts_tablet_manager.cc:1918] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:44.127002 18061 log.cc:1196] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/a34d9664eccb4ea1afea8f347de6fd4b
I20260620 18:34:44.127282 18061 ts_tablet_manager.cc:1939] T a34d9664eccb4ea1afea8f347de6fd4b P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting consensus metadata
W20260620 18:34:44.127699 17897 connection.cc:441] server connection from 127.0.0.1:33432 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33432 (request call id 1) could send its response
I20260620 18:34:44.128666 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.122202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.10.109.124:42029)
0620 18:34:44.122237 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.122239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.122246 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:34:44.122264 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.122265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.122275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.122328 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.123020 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.123493 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.124433 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.124659 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.124738 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.125935 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.125936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.125937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.125943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.127400 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.127882 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.127883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.127883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.127950 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.128153 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.128154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.128155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.128267 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.128374 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.128476 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.128554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:34:44.130952 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:44.121988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.10.109.125:45667)
0620 18:34:44.122033 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:44.122035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:44.122043 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:44.122063 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:44.122064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:44.122075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:44.122124 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:44.123369 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.124112 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.125201 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:44.125523 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:44.125562 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:44.128346 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:44.128347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:44.128348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:44.128354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:44.129918 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.130228 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.130228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.130229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.130284 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:44.130469 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:44.130469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:44.130470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:44.130595 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:44.130688 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:44.130782 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:34:44.130863 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:45.108415 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.108047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:50593)
0620 18:34:45.108269 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.108353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
W20260620 18:34:45.108817 18010 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:43923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.108764 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.108484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:43431)
0620 18:34:45.108617 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.108722 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
W20260620 18:34:45.109033 17737 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:43923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.109273 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.108988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:50529)
0620 18:34:45.109144 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.109226 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":53,"threads_started":1}
W20260620 18:34:45.109453 17874 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:43923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.612192 18074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a673faa5b894849b1cfe935680c6f45)
I20260620 18:34:45.612308 18074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:45.612564 18074 leader_election.cc:288] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124:42029)
I20260620 18:34:45.612581 18075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a673faa5b894849b1cfe935680c6f45)
I20260620 18:34:45.612697 18075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:45.612895 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.612658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40190)
0620 18:34:45.612793 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.612834 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260620 18:34:45.612908 18075 leader_election.cc:288] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125:45667)
W20260620 18:34:45.613001 17349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.613335 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.613018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40202)
0620 18:34:45.613114 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.613159 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
W20260620 18:34:45.613539 17542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.613606 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.613515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40210)
0620 18:34:45.613552 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.613575 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 18:34:45.613678 17349 leader_election.cc:334] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.614014 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.613908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40224)
0620 18:34:45.613960 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.613990 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 18:34:45.614127 17542 leader_election.cc:334] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.616154 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.613160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:45667 (local address 127.0.0.1:45366)
0620 18:34:45.613230 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.613239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:45.613289 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:45.613529 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:45.613529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:45.613531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:45.613583 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:45.613585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.614501 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.614502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:45.615326 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:45.615327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.615905 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:45.615963 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:45.615965 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:45.616016 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:34:45.616076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"thread_start_us":37,"threads_started":1}
I20260620 18:34:45.616155 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.612829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.10.109.124:42029)
0620 18:34:45.612916 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:45.612918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:45.613143 (+ 225us) server_negotiation.cc:407] Connection header received
0620 18:34:45.613169 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:45.613169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:45.613180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:45.613234 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:45.613470 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.614253 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615284 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.615816 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615918 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:45.615927 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:45.616067 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:34:45.616097 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":42,"threads_started":1}
I20260620 18:34:45.616181 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.613210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.10.109.125:45667)
0620 18:34:45.613380 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:45.613382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:45.613390 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:45.613412 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:45.613413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:45.613431 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:45.613646 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:45.613679 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.614427 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615466 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.615815 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615897 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:45.615906 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:45.616036 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:34:45.616076 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260620 18:34:45.616171 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.612698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42029 (local address 127.0.0.1:53588)
0620 18:34:45.613009 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.613020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:45.613085 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:45.613286 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:45.613287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:45.613289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:45.613347 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:45.613350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.614331 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.614332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:45.615204 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:45.615207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:45.615941 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:45.615946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:45.615985 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:45.615988 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:45.616056 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:34:45.616087 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":60,"threads_started":1}
I20260620 18:34:45.616430 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93d350436b7d4ff48289c2db754a4e72" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5faca9ffaeb1442589a58040c0b33b04" is_pre_election: true
I20260620 18:34:45.616537 17373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93d350436b7d4ff48289c2db754a4e72 in term 1.
I20260620 18:34:45.616708 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5faca9ffaeb1442589a58040c0b33b04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93d350436b7d4ff48289c2db754a4e72" is_pre_election: true
I20260620 18:34:45.616799 17567 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5faca9ffaeb1442589a58040c0b33b04 in term 1.
I20260620 18:34:45.616904 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5faca9ffaeb1442589a58040c0b33b04, 93d350436b7d4ff48289c2db754a4e72; no voters: 1a673faa5b894849b1cfe935680c6f45
I20260620 18:34:45.616933 17351 leader_election.cc:302] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5faca9ffaeb1442589a58040c0b33b04, 93d350436b7d4ff48289c2db754a4e72; no voters: 1a673faa5b894849b1cfe935680c6f45
I20260620 18:34:45.617010 18075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:34:45.617043 18075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a673faa5b894849b1cfe935680c6f45)
I20260620 18:34:45.617053 18074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:34:45.617074 18075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:34:45.617086 18074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a673faa5b894849b1cfe935680c6f45)
I20260620 18:34:45.617120 18074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:34:45.617753 18075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:45.617753 18074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:45.617882 18075 leader_election.cc:288] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 election: Requested vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125:45667)
I20260620 18:34:45.617887 18074 leader_election.cc:288] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 election: Requested vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 93d350436b7d4ff48289c2db754a4e72 (127.10.109.124:42029)
I20260620 18:34:45.618031 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.617955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40240)
0620 18:34:45.617982 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.618007 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 18:34:45.618085 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93d350436b7d4ff48289c2db754a4e72" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5faca9ffaeb1442589a58040c0b33b04"
I20260620 18:34:45.618134 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.618008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40256)
0620 18:34:45.618087 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.618112 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:34:45.618207 17373 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93d350436b7d4ff48289c2db754a4e72 in current term 2: Already voted for candidate 5faca9ffaeb1442589a58040c0b33b04 in this term.
I20260620 18:34:45.618338 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5faca9ffaeb1442589a58040c0b33b04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93d350436b7d4ff48289c2db754a4e72"
I20260620 18:34:45.618417 17567 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5faca9ffaeb1442589a58040c0b33b04 in current term 2: Already voted for candidate 93d350436b7d4ff48289c2db754a4e72 in this term.
I20260620 18:34:45.618531 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.618454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40264)
0620 18:34:45.618489 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.618511 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 18:34:45.618590 17349 leader_election.cc:334] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.618628 17349 leader_election.cc:302] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5faca9ffaeb1442589a58040c0b33b04; no voters: 1a673faa5b894849b1cfe935680c6f45, 93d350436b7d4ff48289c2db754a4e72
I20260620 18:34:45.618655 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:45.618578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40266)
0620 18:34:45.618613 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:45.618634 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 18:34:45.618680 18074 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 18:34:45.618712 17542 leader_election.cc:334] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:45.618764 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93d350436b7d4ff48289c2db754a4e72; no voters: 1a673faa5b894849b1cfe935680c6f45, 5faca9ffaeb1442589a58040c0b33b04
I20260620 18:34:45.618813 18075 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:34:46.109416 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:46.109119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:59369)
0620 18:34:46.109306 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:46.109356 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260620 18:34:46.109515 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:46.109311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:48235)
0620 18:34:46.109426 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:46.109463 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260620 18:34:46.110054 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:46.109842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:45707)
0620 18:34:46.109991 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:46.110015 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":43,"threads_started":1}
I20260620 18:34:47.110458 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.109999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:53479)
0620 18:34:47.110349 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.110396 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
I20260620 18:34:47.110447 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.110232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:55995)
0620 18:34:47.110338 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.110383 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260620 18:34:47.110780 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.110523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:42931)
0620 18:34:47.110668 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.110715 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260620 18:34:47.253625 18090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:47.253719 18090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:47.253851 18090 leader_election.cc:288] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125:45667)
I20260620 18:34:47.254232 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.254027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40464)
0620 18:34:47.254143 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.254185 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260620 18:34:47.254315 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93d350436b7d4ff48289c2db754a4e72" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5faca9ffaeb1442589a58040c0b33b04" is_pre_election: true
I20260620 18:34:47.254441 17373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93d350436b7d4ff48289c2db754a4e72 in term 2.
I20260620 18:34:47.254693 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5faca9ffaeb1442589a58040c0b33b04, 93d350436b7d4ff48289c2db754a4e72; no voters:
I20260620 18:34:47.254733 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.254627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40476)
0620 18:34:47.254684 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.254712 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 18:34:47.254798 17542 leader_election.cc:334] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:47.254814 18090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 18:34:47.254885 18090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:47.254920 18090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:34:47.255652 18090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:47.255785 18090 leader_election.cc:288] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 election: Requested vote from peers 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923), 5faca9ffaeb1442589a58040c0b33b04 (127.10.109.125:45667)
I20260620 18:34:47.256035 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.255889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40492)
0620 18:34:47.255989 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.256012 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:34:47.256064 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93d350436b7d4ff48289c2db754a4e72" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5faca9ffaeb1442589a58040c0b33b04"
I20260620 18:34:47.256170 17373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:34:47.256418 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.256302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40498)
0620 18:34:47.256372 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.256397 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:34:47.256522 17542 leader_election.cc:334] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
I20260620 18:34:47.256831 17373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93d350436b7d4ff48289c2db754a4e72 in term 3.
I20260620 18:34:47.257014 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5faca9ffaeb1442589a58040c0b33b04, 93d350436b7d4ff48289c2db754a4e72; no voters: 1a673faa5b894849b1cfe935680c6f45
I20260620 18:34:47.257105 18090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:34:47.257226 18090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [term 3 LEADER]: Becoming Leader. State: Replica: 93d350436b7d4ff48289c2db754a4e72, State: Running, Role: LEADER
I20260620 18:34:47.257313 18090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } }
I20260620 18:34:47.257692 18093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d350436b7d4ff48289c2db754a4e72. Latest consensus state: current_term: 3 leader_uuid: "93d350436b7d4ff48289c2db754a4e72" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:47.257788 18093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:47.257977 18095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:47.258332 18095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:47.258383 18095 catalog_manager.cc:1259] Loaded cluster ID: a335d32e3b494575aaee5e639a2be3e6
I20260620 18:34:47.258399 18095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:47.258529 18095 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:47.258579 18095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Loaded TSK: 0
I20260620 18:34:47.258663 18095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:34:47.295811 17556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:47.297380 17556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:47.298338 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.298243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40526)
0620 18:34:47.298289 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.298317 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:34:47.298342 17373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [term 3 FOLLOWER]: Refusing update from remote peer 93d350436b7d4ff48289c2db754a4e72: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 18:34:47.298676 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.298573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.0.0.1:40528)
0620 18:34:47.298623 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.298650 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:34:47.298709 18090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 18:34:47.298849 17542 consensus_peers.cc:487] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 -> Peer 1a673faa5b894849b1cfe935680c6f45 (127.10.109.126:43923): Couldn't send request to peer 1a673faa5b894849b1cfe935680c6f45. Status: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:34:47.299383 18097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d350436b7d4ff48289c2db754a4e72. Latest consensus state: current_term: 3 leader_uuid: "93d350436b7d4ff48289c2db754a4e72" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:47.299523 18097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:47.299943 18093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "93d350436b7d4ff48289c2db754a4e72" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:47.300061 18093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:47.300886 18098 mvcc.cc:205] Tried to move back new op lower bound from 7298992075972087808 to 7298992075806752768. Current Snapshot: MvccSnapshot[applied={T|T < 7298992075972087808}]
I20260620 18:34:47.300846 18097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "93d350436b7d4ff48289c2db754a4e72" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1a673faa5b894849b1cfe935680c6f45" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43923 } } peers { permanent_uuid: "5faca9ffaeb1442589a58040c0b33b04" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 45667 } attrs { promote: false } } peers { permanent_uuid: "93d350436b7d4ff48289c2db754a4e72" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42029 } attrs { promote: false } } }
I20260620 18:34:47.300942 18097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faca9ffaeb1442589a58040c0b33b04 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:34:47.305289 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33709 (local address 127.0.0.1:40268)
0620 18:34:47.302379 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.302389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:47.302439 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:47.302618 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:47.302619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:47.302621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:47.302669 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.302671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303307 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.303308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.304448 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.304450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.305054 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.305115 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305120 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:47.305170 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 18:34:47.305211 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":156,"thread_start_us":56,"threads_started":1}
I20260620 18:34:47.305476 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.10.109.67:34819)
0620 18:34:47.302531 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.302533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.302539 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:34:47.302562 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.302563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.302573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.302639 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:47.302778 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303519 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.304515 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.305070 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305159 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.305261 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.305307 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260620 18:34:47.305505 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.10.109.65:33709)
0620 18:34:47.302265 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.302267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.302433 (+ 166us) server_negotiation.cc:407] Connection header received
0620 18:34:47.302490 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.302492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.302505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.302570 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:47.302723 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303223 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.304538 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.304974 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305055 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305067 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.305391 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.305444 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260620 18:34:47.305596 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37641 (local address 127.0.0.1:38096)
0620 18:34:47.302247 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.302259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:47.302321 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:47.302549 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:47.302550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:47.302552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:47.302606 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.302608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303551 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.303551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.304606 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.304608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.305330 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.305399 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305404 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:34:47.305438 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 18:34:47.305477 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":26}
I20260620 18:34:47.305693 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.10.109.66:37641)
0620 18:34:47.302207 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.302209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.302305 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:34:47.302441 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.302442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.302453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.302668 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:47.302705 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303472 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.304683 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.305242 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305522 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305537 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.305558 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.305608 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260620 18:34:47.305888 17670 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 (DEFAULT_TABLE table=test-table [id=46015755ce5d4b5693bd1a0100b75a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:47.305934 17807 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 (DEFAULT_TABLE table=test-table [id=46015755ce5d4b5693bd1a0100b75a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:47.306011 17670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7dd39c0ecb46ada734fe180c90f7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:47.306074 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7dd39c0ecb46ada734fe180c90f7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:47.307276 18107 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: Bootstrap starting.
I20260620 18:34:47.307590 18108 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Bootstrap starting.
I20260620 18:34:47.307641 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.302111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34819 (local address 127.0.0.1:33444)
0620 18:34:47.302283 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.302294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:47.302346 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:47.302642 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:47.302643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:47.302645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:47.302679 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.302680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.303599 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.303600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.304431 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.304433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.305147 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.305148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.305200 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.305203 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:47.305247 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 18:34:47.305277 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260620 18:34:47.307770 18107 tablet_bootstrap.cc:654] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:47.307997 18108 tablet_bootstrap.cc:654] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:47.307992 17943 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 (DEFAULT_TABLE table=test-table [id=46015755ce5d4b5693bd1a0100b75a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:47.308094 17943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7dd39c0ecb46ada734fe180c90f7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:47.308391 18107 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: No bootstrap required, opened a new log
I20260620 18:34:47.308434 18107 ts_tablet_manager.cc:1397] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:47.308545 18107 raft_consensus.cc:348] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.308609 18107 raft_consensus.cc:374] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:47.308622 18107 raft_consensus.cc:729] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f391411f1fe24c91864a8e92daf750fd, State: Initialized, Role: FOLLOWER
I20260620 18:34:47.308691 18107 consensus_queue.cc:260] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.308857 18107 ts_tablet_manager.cc:1428] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:47.309226 18108 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: No bootstrap required, opened a new log
I20260620 18:34:47.309283 18108 ts_tablet_manager.cc:1397] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:47.309456 17736 heartbeater.cc:503] Master 127.10.109.124:42029 was elected leader, sending a full tablet report...
I20260620 18:34:47.309424 18108 raft_consensus.cc:348] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.309511 18108 raft_consensus.cc:374] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:47.309535 18108 raft_consensus.cc:729] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4560515d11024407b702f9acb1e19eff, State: Initialized, Role: FOLLOWER
I20260620 18:34:47.309617 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.309030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:45033)
0620 18:34:47.309555 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.309587 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504}
I20260620 18:34:47.309774 18108 consensus_queue.cc:260] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.309891 18112 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Bootstrap starting.
I20260620 18:34:47.309973 18108 ts_tablet_manager.cc:1428] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:47.310031 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.309918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:40693)
0620 18:34:47.309982 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.310004 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:34:47.310228 18112 tablet_bootstrap.cc:654] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:47.310240 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.310142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:39007)
0620 18:34:47.310186 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.310212 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 18:34:47.310824 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.310681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:34917)
0620 18:34:47.310757 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.310796 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:34:47.310966 18112 tablet_bootstrap.cc:492] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: No bootstrap required, opened a new log
I20260620 18:34:47.311017 18112 ts_tablet_manager.cc:1397] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:47.311147 18112 raft_consensus.cc:348] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.311210 18112 raft_consensus.cc:374] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:47.311224 18112 raft_consensus.cc:729] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86f7bc8eafc435bb9d374abf90ad5bd, State: Initialized, Role: FOLLOWER
I20260620 18:34:47.311277 18112 consensus_queue.cc:260] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.311440 18112 ts_tablet_manager.cc:1428] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:47.311669 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.311583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:47509)
0620 18:34:47.311616 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.311646 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 18:34:47.311822 18009 heartbeater.cc:503] Master 127.10.109.124:42029 was elected leader, sending a full tablet report...
I20260620 18:34:47.312062 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.311980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:50875)
0620 18:34:47.312015 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.312041 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 18:34:47.379555 17876 tablet.cc:2369] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:47.402156 18113 raft_consensus.cc:482] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:47.402244 18113 raft_consensus.cc:504] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.402621 18113 leader_election.cc:288] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709), f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:47.406081 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.402873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:34819 (local address 127.10.109.66:39451)
0620 18:34:47.402925 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.402937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:47.402995 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:47.403130 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:47.403131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:47.403134 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:47.403223 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.403225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.404305 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.404306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.405212 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.405214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.405780 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.405852 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.405894 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:34:47.405937 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 18:34:47.406001 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":79}
I20260620 18:34:47.406092 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.402947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:39451 (local address 127.10.109.67:34819)
0620 18:34:47.402992 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.402994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.403004 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:47.403056 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.403057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.403066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.403117 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:47.403409 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.404113 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405306 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.405697 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405752 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.405759 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.405971 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.406019 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:47.406111 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.403014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:49189 (local address 127.10.109.65:33709)
0620 18:34:47.403063 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.403065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.403173 (+ 108us) server_negotiation.cc:407] Connection header received
0620 18:34:47.403223 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.403223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.403234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.403293 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:47.403596 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.404427 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405383 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.405748 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405819 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.405828 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.406009 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.406047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:34:47.406163 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.402937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33709 (local address 127.10.109.66:49189)
0620 18:34:47.403107 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.403115 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:47.403164 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:47.403407 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:47.403408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:47.403410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:47.403470 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.403472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.404524 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.404525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.405297 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:47.405300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.405855 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.405856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:47.405905 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.405909 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:47.405989 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:34:47.406024 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260620 18:34:47.406461 17963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7dd39c0ecb46ada734fe180c90f7e1" candidate_uuid: "4560515d11024407b702f9acb1e19eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" is_pre_election: true
I20260620 18:34:47.406562 17963 raft_consensus.cc:2454] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4560515d11024407b702f9acb1e19eff in term 0.
I20260620 18:34:47.406620 17690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7dd39c0ecb46ada734fe180c90f7e1" candidate_uuid: "4560515d11024407b702f9acb1e19eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f391411f1fe24c91864a8e92daf750fd" is_pre_election: true
I20260620 18:34:47.406759 17690 raft_consensus.cc:2454] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4560515d11024407b702f9acb1e19eff in term 0.
I20260620 18:34:47.406961 17761 leader_election.cc:302] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4560515d11024407b702f9acb1e19eff, f391411f1fe24c91864a8e92daf750fd; no voters:
I20260620 18:34:47.407219 18113 raft_consensus.cc:2781] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:47.407320 18113 raft_consensus.cc:482] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:47.407346 18113 raft_consensus.cc:3037] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:47.408346 18113 raft_consensus.cc:504] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.408627 18113 leader_election.cc:288] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [CANDIDATE]: Term 1 election: Requested vote from peers f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709), f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819)
I20260620 18:34:47.408818 17963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7dd39c0ecb46ada734fe180c90f7e1" candidate_uuid: "4560515d11024407b702f9acb1e19eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd"
I20260620 18:34:47.408887 17963 raft_consensus.cc:3037] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:47.408792 17690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7dd39c0ecb46ada734fe180c90f7e1" candidate_uuid: "4560515d11024407b702f9acb1e19eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f391411f1fe24c91864a8e92daf750fd"
I20260620 18:34:47.408926 17690 raft_consensus.cc:3037] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:47.409713 17963 raft_consensus.cc:2454] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4560515d11024407b702f9acb1e19eff in term 1.
I20260620 18:34:47.409737 17690 raft_consensus.cc:2454] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4560515d11024407b702f9acb1e19eff in term 1.
I20260620 18:34:47.410058 17761 leader_election.cc:302] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4560515d11024407b702f9acb1e19eff, f391411f1fe24c91864a8e92daf750fd; no voters:
I20260620 18:34:47.410167 18113 raft_consensus.cc:2781] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:47.410297 18113 raft_consensus.cc:686] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 1 LEADER]: Becoming Leader. State: Replica: 4560515d11024407b702f9acb1e19eff, State: Running, Role: LEADER
I20260620 18:34:47.410398 17873 heartbeater.cc:503] Master 127.10.109.124:42029 was elected leader, sending a full tablet report...
I20260620 18:34:47.410396 18113 consensus_queue.cc:237] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } }
I20260620 18:34:47.410800 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.410548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:51167)
0620 18:34:47.410709 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.410765 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 18:34:47.411269 17557 catalog_manager.cc:5495] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff reported cstate change: term changed from 0 to 1, leader changed from <none> to 4560515d11024407b702f9acb1e19eff (127.10.109.66). New cstate: current_term: 1 leader_uuid: "4560515d11024407b702f9acb1e19eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4560515d11024407b702f9acb1e19eff" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:47.422318 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.419788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.10.109.66:37641)
0620 18:34:47.419888 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:47.419890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:47.419896 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:34:47.419924 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:47.419924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:47.419942 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:47.420121 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:47.420273 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.420778 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.421546 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:47.421731 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:47.421834 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:47.421995 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:47.422095 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:47.422178 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:47.422219 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 18:34:47.422239 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:34:47.423458 17963 raft_consensus.cc:1261] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Refusing update from remote peer 4560515d11024407b702f9acb1e19eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:47.423642 17690 raft_consensus.cc:1261] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Refusing update from remote peer 4560515d11024407b702f9acb1e19eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:47.423751 18117 consensus_queue.cc:1035] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86f7bc8eafc435bb9d374abf90ad5bd" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:47.424073 18117 consensus_queue.cc:1035] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f391411f1fe24c91864a8e92daf750fd" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:47.424649 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.424529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:39893)
0620 18:34:47.424587 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.424620 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:34:47.425812 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.425691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:56805)
0620 18:34:47.425755 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.425780 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:34:47.425933 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.425767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:41217)
0620 18:34:47.425878 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.425903 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:34:47.426839 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.426725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.67:59497)
0620 18:34:47.426779 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.426809 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:34:47.427331 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.427251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.66:46659)
0620 18:34:47.427281 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.427306 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 18:34:47.427762 17557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:47.427848 17557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:47.428170 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:47.427873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43923 (local address 127.10.109.65:34163)
0620 18:34:47.428110 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.428140 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260620 18:34:47.429246 17557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93d350436b7d4ff48289c2db754a4e72: Sending DeleteTablet for 3 replicas of tablet 4e7dd39c0ecb46ada734fe180c90f7e1
I20260620 18:34:47.429508 17943 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:47 UTC) from {username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:33444
I20260620 18:34:47.429590 17807 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:47 UTC) from {username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:38096
I20260620 18:34:47.429714 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7dd39c0ecb46ada734fe180c90f7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:47 UTC) from {username='oryx', principal='oryx/127.10.109.124@KRBTEST.COM'} at 127.0.0.1:40268
I20260620 18:34:47.429904 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17511
I20260620 18:34:47.430111 18131 tablet_replica.cc:330] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd: stopping tablet replica
I20260620 18:34:47.430168 18131 raft_consensus.cc:2229] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:47.430210 18131 raft_consensus.cc:2258] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f391411f1fe24c91864a8e92daf750fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:47.431774 18130 tablet_replica.cc:330] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: stopping tablet replica
I20260620 18:34:47.431854 18130 raft_consensus.cc:2229] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:47.431938 18130 raft_consensus.cc:2258] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:47.432061 18129 tablet_replica.cc:330] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: stopping tablet replica
I20260620 18:34:47.432117 18129 raft_consensus.cc:2229] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:47.432166 18129 raft_consensus.cc:2258] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:47.432437 18130 ts_tablet_manager.cc:1905] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:47.433823 18129 ts_tablet_manager.cc:1905] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:47.434136 18130 ts_tablet_manager.cc:1918] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:47.434247 18130 log.cc:1196] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/4e7dd39c0ecb46ada734fe180c90f7e1
I20260620 18:34:47.434561 18130 ts_tablet_manager.cc:1939] T 4e7dd39c0ecb46ada734fe180c90f7e1 P 4560515d11024407b702f9acb1e19eff: Deleting consensus metadata
I20260620 18:34:47.434970 17543 catalog_manager.cc:4841] TS 4560515d11024407b702f9acb1e19eff (127.10.109.66:37641): tablet 4e7dd39c0ecb46ada734fe180c90f7e1 (table test-table [id=46015755ce5d4b5693bd1a0100b75a3a]) successfully deleted
I20260620 18:34:47.435472 18129 ts_tablet_manager.cc:1918] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:47.435544 18129 log.cc:1196] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/4e7dd39c0ecb46ada734fe180c90f7e1
I20260620 18:34:47.435750 18129 ts_tablet_manager.cc:1939] T 4e7dd39c0ecb46ada734fe180c90f7e1 P f86f7bc8eafc435bb9d374abf90ad5bd: Deleting consensus metadata
I20260620 18:34:47.436110 17543 catalog_manager.cc:4841] TS f86f7bc8eafc435bb9d374abf90ad5bd (127.10.109.67:34819): tablet 4e7dd39c0ecb46ada734fe180c90f7e1 (table test-table [id=46015755ce5d4b5693bd1a0100b75a3a]) successfully deleted
W20260620 18:34:47.438583 18104 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:47.438451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33709 (local address 127.0.0.1:40284)
0620 18:34:47.438506 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:47.438534 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33709: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:34:47.438719 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17744
W20260620 18:34:47.438798 17542 catalog_manager.cc:4584] TS f391411f1fe24c91864a8e92daf750fd (127.10.109.65:33709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e7dd39c0ecb46ada734fe180c90f7e1: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33709: connect: Connection reset by peer (error 104)
I20260620 18:34:47.444165 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17880
I20260620 18:34:47.448599 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17230
I20260620 18:34:47.453423 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 17418
2026-06-20T18:34:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6167 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:47 dist-test-slave-3jql krb5kdc[18134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:47 dist-test-slave-3jql krb5kdc[18134](info): set up 2 sockets
Jun 20 18:34:47 dist-test-slave-3jql krb5kdc[18134](info): commencing operation
krb5kdc: starting...
W20260620 18:34:48.503521 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:48 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab.
Entry for principal oryx/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab.
I20260620 18:34:48.621354 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42279
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab
--principal=oryx/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:48.703193 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:48.703374 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:48.703397 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:48.704516 18147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:48.704557 18147 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:34:48.704571 18147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:48.704583 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:48.704595 18147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:48.704607 18147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:48.706100 18147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:42279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.126.keytab
--principal=oryx/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:48.706326 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:48.706584 18147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:48.711375 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:48.711391 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:48.711381 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:48.711607 18147 server_base.cc:1034] running on GCE node
I20260620 18:34:48.711894 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:48.712162 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:48.713331 18147 hybrid_clock.cc:648] HybridClock initialized: now 1781980488713297 us; error 63 us; skew 500 ppm
Jun 20 18:34:48 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:48.716555 18147 init.cc:377] Logged in from keytab as oryx/127.10.109.126@KRBTEST.COM (short username oryx)
I20260620 18:34:48.717168 18147 webserver.cc:460] Webserver started at http://127.10.109.126:44423/ using document root <none> and password file <none>
I20260620 18:34:48.717417 18147 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:48.717465 18147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:48.717576 18147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:48.718806 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "fe53db8204874e57998c4fafe53056ef"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "f095e27717dfa92729f1c181feade7c7"
server_key_iv: "0350876f5c9edf0e3c75312f135f3f34"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.719383 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "fe53db8204874e57998c4fafe53056ef"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "f095e27717dfa92729f1c181feade7c7"
server_key_iv: "0350876f5c9edf0e3c75312f135f3f34"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.720741 18147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:48.721583 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:48.721846 18147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:48.721918 18147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "fe53db8204874e57998c4fafe53056ef"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "f095e27717dfa92729f1c181feade7c7"
server_key_iv: "0350876f5c9edf0e3c75312f135f3f34"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.722030 18147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:48.746446 18147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:48.749104 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:48.749318 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:48.755108 18147 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:42279
I20260620 18:34:48.755136 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:42279 every 8 connection(s)
I20260620 18:34:48.755497 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:48.756261 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:48.756353 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18147
I20260620 18:34:48.756461 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:48.756695 10677 external_mini_cluster.cc:1428] Setting key dabfc85d3df5830d03dbebabd487cded
I20260620 18:34:48.759629 18225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Bootstrap starting.
I20260620 18:34:48.760249 18225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:48.760532 18225 log.cc:824] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:48.761250 18225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: No bootstrap required, opened a new log
I20260620 18:34:48.763082 18225 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } }
Jun 20 18:34:48 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:48.763236 18225 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:48.763283 18225 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe53db8204874e57998c4fafe53056ef, State: Initialized, Role: FOLLOWER
I20260620 18:34:48.763417 18225 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } }
I20260620 18:34:48.763497 18225 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:48.763540 18225 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:48.763585 18225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:48.764308 18225 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } }
I20260620 18:34:48.764446 18225 leader_election.cc:302] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe53db8204874e57998c4fafe53056ef; no voters:
I20260620 18:34:48.764607 18225 leader_election.cc:288] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:48.764762 18230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:48.764878 18225 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:48.764940 18230 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [term 1 LEADER]: Becoming Leader. State: Replica: fe53db8204874e57998c4fafe53056ef, State: Running, Role: LEADER
I20260620 18:34:48.765120 18230 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } }
I20260620 18:34:48.765483 18231 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe53db8204874e57998c4fafe53056ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } } }
I20260620 18:34:48.765581 18231 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:48.765725 18231 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe53db8204874e57998c4fafe53056ef. Latest consensus state: current_term: 1 leader_uuid: "fe53db8204874e57998c4fafe53056ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe53db8204874e57998c4fafe53056ef" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42279 } } }
I20260620 18:34:48.765784 18231 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef [sys.catalog]: This master's current role is: LEADER
W20260620 18:34:48.766726 18246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:34:48.766816 18246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:34:48.766873 18239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:48.767006 18239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:48.768466 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.757219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.10.109.126:42279)
0620 18:34:48.757403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:48.757418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:48.757443 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:48.757943 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:48.757950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:48.757994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:48.758105 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:48.758866 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.759828 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.760681 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.761231 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.761347 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.763787 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:48.763791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:48.763791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:48.763803 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:48.766724 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.767159 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.767162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.767163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.767261 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.767489 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.767490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.767490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.767744 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:48.767919 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:48.768116 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:34:48.768265 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260620 18:34:48.768550 18239 catalog_manager.cc:1347] Generated new cluster ID: 7757211fff75485896b9f74997c85db1
I20260620 18:34:48.768584 18239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:48.778537 18239 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:48.779155 18239 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:48.786324 18239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Generated new TSK 0
I20260620 18:34:48.786520 18239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
I20260620 18:34:48.818617 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42279
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:48.899509 18251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:48.899734 18251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:48.899770 18251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:48.900918 18251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:48.900985 18251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:48.901021 18251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:48.901058 18251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:48.902508 18251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:48.902809 18251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:48.903046 18251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:48.908037 18257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:48.908224 18256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:48.908372 18251 server_base.cc:1034] running on GCE node
W20260620 18:34:48.908397 18259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:48.908618 18251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:48.908807 18251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:48.909986 18251 hybrid_clock.cc:648] HybridClock initialized: now 1781980488909974 us; error 39 us; skew 500 ppm
Jun 20 18:34:48 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:48.912765 18251 init.cc:377] Logged in from keytab as oryx/127.10.109.65@KRBTEST.COM (short username oryx)
I20260620 18:34:48.913192 18251 webserver.cc:460] Webserver started at http://127.10.109.65:35687/ using document root <none> and password file <none>
I20260620 18:34:48.913424 18251 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:48.913491 18251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:48.913641 18251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:48.914657 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "5640511b999842d2891b6af2b5f7a39b"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "34e21b5b4af26bcd31a66af5d18c5677"
server_key_iv: "2738a811dbe1dc316789eb841db02a7a"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.915081 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "5640511b999842d2891b6af2b5f7a39b"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "34e21b5b4af26bcd31a66af5d18c5677"
server_key_iv: "2738a811dbe1dc316789eb841db02a7a"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.916419 18251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:48.917240 18266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:48.917413 18251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:48.917518 18251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "5640511b999842d2891b6af2b5f7a39b"
format_stamp: "Formatted at 2026-06-20 18:34:48 on dist-test-slave-3jql"
server_key: "34e21b5b4af26bcd31a66af5d18c5677"
server_key_iv: "2738a811dbe1dc316789eb841db02a7a"
server_key_version: "encryptionkey@0"
I20260620 18:34:48.917601 18251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:48.932408 18251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:48.934696 18251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:48.934917 18251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:48.935456 18251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:48.935521 18251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:48.935565 18251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:48.935607 18251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:48 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:48.942682 18251 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:34355
I20260620 18:34:48.943045 18251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:48.943220 18381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:34355 every 8 connection(s)
I20260620 18:34:48.943815 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18251
I20260620 18:34:48.943949 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:48.944154 10677 external_mini_cluster.cc:1428] Setting key 1ec8317160d841e71b8c40dffba67c5d
I20260620 18:34:48.946175 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.936046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:39369 (local address 127.10.109.126:42279)
0620 18:34:48.936108 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:48.936111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:48.936299 (+ 188us) server_negotiation.cc:407] Connection header received
0620 18:34:48.936918 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:48.936919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:48.936934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:48.937005 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:48.938014 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.938889 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.940079 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.940302 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.940387 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.943127 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:48.943128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:48.943128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:48.943136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:48.944728 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.945189 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.945190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.945191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.945265 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.945538 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.945539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.945539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.945658 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:48.945733 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:48.946000 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:34:48.946084 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:34:48.946369 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.935989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.65:39369)
0620 18:34:48.936182 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:48.936216 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:48.936761 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:48.937086 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:48.937091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:48.937484 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:48.937895 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:48.937903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.938989 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.938990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:48.939966 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:48.939969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.940413 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.940414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:48.940545 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.941573 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:48.941581 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:48.943028 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:48.944827 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:48.944834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:48.944836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:48.945101 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:48.945350 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:48.945351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:48.945351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:48.945463 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:48.945726 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:48.945733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:48.945869 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 18:34:48.945979 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:34:48.946135 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260620 18:34:48.946883 18382 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42279
I20260620 18:34:48.946965 18382 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:48.947149 18382 heartbeater.cc:511] Master 127.10.109.126:42279 requested a full tablet report, sending...
I20260620 18:34:48.947690 18178 ts_manager.cc:194] Registered new tserver with Master: 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355)
I20260620 18:34:48.948388 18178 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:39369
WARNING: no policy specified for oryx/127.10.109.66@KRBTEST.COM; defaulting to no policy
I20260620 18:34:48.954960 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.947968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:48677 (local address 127.10.109.126:42279)
0620 18:34:48.948003 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:48.948005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:48.948729 (+ 724us) server_negotiation.cc:407] Connection header received
0620 18:34:48.948771 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:48.948772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:48.948783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:48.948834 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:48.949168 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.949903 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.950617 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.950993 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.951069 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.951957 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:48.951958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:48.951958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:48.951965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:48.953722 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.954098 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.954099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.954100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.954166 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:48.954413 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:48.954414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:48.954415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:48.954533 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:48.954608 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:48.954785 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:34:48.954871 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:34:48.954990 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:48.947901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.65:48677)
0620 18:34:48.948638 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:48.948650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:48.948716 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:48.948846 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:48.948846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:48.949041 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:48.949092 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:48.949093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.950009 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.950010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:48.950527 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:48.950529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:48.951079 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:48.951080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:48.951131 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:48.951603 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:48.951607 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:48.951870 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:48.953816 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:48.953817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:48.953818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:48.954021 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:48.954237 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:48.954237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:48.954238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:48.954301 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:48.954610 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:48.954611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:48.954661 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:34:48.954735 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:34:48.954894 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":64,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
I20260620 18:34:48.974269 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42279
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:49.063687 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:49.063928 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:49.063967 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:49.065212 18391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:49.065282 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:49.065320 18391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:49.065363 18391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:49.067087 18391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:49.067332 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:49.067575 18391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:49.072557 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.072767 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.072830 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:49.073153 18391 server_base.cc:1034] running on GCE node
I20260620 18:34:49.073457 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:49.073710 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:49.074833 18391 hybrid_clock.cc:648] HybridClock initialized: now 1781980489074820 us; error 26 us; skew 500 ppm
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:49.077522 18391 init.cc:377] Logged in from keytab as oryx/127.10.109.66@KRBTEST.COM (short username oryx)
I20260620 18:34:49.077993 18391 webserver.cc:460] Webserver started at http://127.10.109.66:41725/ using document root <none> and password file <none>
I20260620 18:34:49.078229 18391 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:49.078295 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:49.078450 18391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:49.079566 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "82eacff391e944b995edbb285209392b"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "8a64a53ea3faf9164e6e9670133d6fbe"
server_key_iv: "297fda04686947df890d6656a26780b9"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.079934 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "82eacff391e944b995edbb285209392b"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "8a64a53ea3faf9164e6e9670133d6fbe"
server_key_iv: "297fda04686947df890d6656a26780b9"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.081241 18391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:34:49.082108 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.082288 18391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.082397 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "82eacff391e944b995edbb285209392b"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "8a64a53ea3faf9164e6e9670133d6fbe"
server_key_iv: "297fda04686947df890d6656a26780b9"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.082499 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:49.090183 18391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:49.092418 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:49.092574 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:49.093043 18391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:49.093106 18391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.093152 18391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:49.093175 18391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:49.100203 18391 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:35479
I20260620 18:34:49.100278 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:35479 every 8 connection(s)
I20260620 18:34:49.100584 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:49.103396 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.093619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:57795 (local address 127.10.109.126:42279)
0620 18:34:49.093667 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.093669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.093867 (+ 198us) server_negotiation.cc:407] Connection header received
0620 18:34:49.094350 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.094351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.094364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.094467 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.095346 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.096205 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.097338 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.097646 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.097718 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.100572 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.100575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.100575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.100583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.101945 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.102346 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.102347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.102347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.102410 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.102706 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.102707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.102707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.102859 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.102955 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.103219 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.103294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:34:49.103578 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.093616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.66:57795)
0620 18:34:49.093757 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.093789 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.094252 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.094541 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.094545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.094944 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:49.095236 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.095242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.096311 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.096312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.097219 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.097222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.097781 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.097782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.097959 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.098534 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:49.098546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:49.100458 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:49.102030 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.102036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.102038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.102250 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.102518 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.102518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.102519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.102632 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.102946 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:49.102952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:49.103089 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.103198 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.103356 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260620 18:34:49.104108 18522 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42279
I20260620 18:34:49.104214 18522 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:49.104436 18522 heartbeater.cc:511] Master 127.10.109.126:42279 requested a full tablet report, sending...
I20260620 18:34:49.104836 18178 ts_manager.cc:194] Registered new tserver with Master: 82eacff391e944b995edbb285209392b (127.10.109.66:35479)
I20260620 18:34:49.105346 18178 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:57795
I20260620 18:34:49.109488 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18391
I20260620 18:34:49.109609 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:49.109871 10677 external_mini_cluster.cc:1428] Setting key a04e8f1489d0d33c6444bc5a39174594
I20260620 18:34:49.110500 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.105134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:50161 (local address 127.10.109.126:42279)
0620 18:34:49.105166 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.105167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.105301 (+ 134us) server_negotiation.cc:407] Connection header received
0620 18:34:49.105325 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.105326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.105336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.105402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.105731 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.106166 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.106882 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.107059 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.107121 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.107835 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.107836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.107837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.107842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.109180 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.109592 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.109593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.109593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.109654 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.109915 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.109915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.109916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.110141 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.110199 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.110343 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.110408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260620 18:34:49.110517 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.105074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.66:50161)
0620 18:34:49.105228 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.105238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.105289 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.105403 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.105404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.105599 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:49.105653 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.105654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.106261 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.106262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.106795 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.106797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.107131 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.107131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.107179 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.107513 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:49.107517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:49.107761 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:49.109257 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.109258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.109258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.109507 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.109768 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.109769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.109769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.109832 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.110209 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:49.110210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:49.110254 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.110317 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.110412 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
I20260620 18:34:49.138243 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:42279
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:49.219573 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:49.219838 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:49.219887 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:49.221148 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:49.221210 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:49.221247 18531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:34:49.221284 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:49.222859 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.109.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:49.223121 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:49.223348 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:49.228693 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.228693 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:49.228675 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:49.228847 18531 server_base.cc:1034] running on GCE node
I20260620 18:34:49.229053 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:49.229288 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:49.230453 18531 hybrid_clock.cc:648] HybridClock initialized: now 1781980489230434 us; error 30 us; skew 500 ppm
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:49.232918 18531 init.cc:377] Logged in from keytab as oryx/127.10.109.67@KRBTEST.COM (short username oryx)
I20260620 18:34:49.233398 18531 webserver.cc:460] Webserver started at http://127.10.109.67:43389/ using document root <none> and password file <none>
I20260620 18:34:49.233621 18531 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:49.233666 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:49.233804 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:49.234905 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "c97624f015c84c859ed28f62c45369fa"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "c591910e169dbde233b34b6f604d5764"
server_key_iv: "52c4d7310404261b93a5b0452d0c1da5"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.235262 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "c97624f015c84c859ed28f62c45369fa"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "c591910e169dbde233b34b6f604d5764"
server_key_iv: "52c4d7310404261b93a5b0452d0c1da5"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.236683 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:49.237452 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.237630 18531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:49.237720 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "c97624f015c84c859ed28f62c45369fa"
format_stamp: "Formatted at 2026-06-20 18:34:49 on dist-test-slave-3jql"
server_key: "c591910e169dbde233b34b6f604d5764"
server_key_iv: "52c4d7310404261b93a5b0452d0c1da5"
server_key_version: "encryptionkey@0"
I20260620 18:34:49.237771 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:49.259589 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:49.261807 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:49.261924 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:49.262395 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:49.262437 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.262462 18531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:49.262511 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:49.269886 18531 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:37807
I20260620 18:34:49.269929 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:37807 every 8 connection(s)
I20260620 18:34:49.270289 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:49.273078 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.262944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39111 (local address 127.10.109.126:42279)
0620 18:34:49.263016 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.263019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.263238 (+ 219us) server_negotiation.cc:407] Connection header received
0620 18:34:49.263765 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.263766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.263777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.263832 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.264667 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.265494 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.267087 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.267430 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.267519 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.270190 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.270191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.270191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.270199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.271620 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.272057 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.272058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.272059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.272122 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.272433 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.272434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.272434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.272557 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.272634 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.272905 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.272986 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:34:49.273284 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.262937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.67:39111)
0620 18:34:49.263102 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.263142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.263664 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.263903 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.263907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.264272 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:49.264544 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.264551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.265593 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.265594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.266991 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.266995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.267532 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.267533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.267680 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.268292 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:49.268303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:49.270098 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:49.271739 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.271746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.271748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.271969 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.272190 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.272191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.272191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.272303 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.272626 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:49.272631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:49.272766 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.272875 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.273046 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260620 18:34:49.273424 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18531
I20260620 18:34:49.273492 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:49.273689 10677 external_mini_cluster.cc:1428] Setting key efbbbb243cb797c8199961454a677d4e
I20260620 18:34:49.273741 18662 heartbeater.cc:347] Connected to a master server at 127.10.109.126:42279
I20260620 18:34:49.273806 18662 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:49.273974 18662 heartbeater.cc:511] Master 127.10.109.126:42279 requested a full tablet report, sending...
I20260620 18:34:49.274309 18179 ts_manager.cc:194] Registered new tserver with Master: c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.274833 18179 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:39111
I20260620 18:34:49.280210 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.274817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:42983 (local address 127.10.109.126:42279)
0620 18:34:49.274855 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.274857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.275027 (+ 170us) server_negotiation.cc:407] Connection header received
0620 18:34:49.275097 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.275098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.275107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.275155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.275506 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.275953 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.276632 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.276819 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.276902 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.277721 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.277722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.277723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.277729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.279130 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.279477 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.279478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.279478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.279531 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.279728 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.279729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.279729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.279836 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.279909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.280060 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.280124 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:34:49.280225 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.274724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.10.109.67:42983)
0620 18:34:49.274952 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.274962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.275041 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.275158 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.275158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.275327 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:49.275386 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.275388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.276045 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.276046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.276537 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.276538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.276904 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.276904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.276973 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.277359 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:49.277364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:49.277617 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:49.279226 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.279226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.279227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.279394 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.279600 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.279601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.279601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.279656 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.279901 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:49.279902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:49.279945 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.280033 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.280126 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":140,"threads_started":1}
I20260620 18:34:49.285118 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:34:49.305291 18228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:49.300050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.10.109.126:42279)
0620 18:34:49.300129 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.300131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.300138 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:34:49.300156 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.300156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.300164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.300227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.300762 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.301233 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.301958 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.302146 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.302219 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.305199 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45042: BlockingRecv error: recv got EOF from 127.0.0.1:45042 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:49.312779 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.306055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.10.109.126:42279)
0620 18:34:49.306106 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.306108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.306168 (+ 60us) server_negotiation.cc:407] Connection header received
0620 18:34:49.306211 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.306211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.306219 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.306265 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.306694 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.307233 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.307861 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.308049 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.308100 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.310202 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.310203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.310203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.310208 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.311455 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.311892 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.311893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.311893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.311946 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.312130 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.312130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.312130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.312229 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.312316 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.312661 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.312720 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 18:34:49.314503 18179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:49.316180 18179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:49.322922 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35479 (local address 127.0.0.1:36210)
0620 18:34:49.318864 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.318904 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.318975 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.319191 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.319194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.319196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.319269 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.319271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320652 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.320652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.321748 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.321750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.322590 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.322668 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.322681 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.322777 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.322831 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":45,"threads_started":1}
I20260620 18:34:49.322934 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.10.109.66:35479)
0620 18:34:49.318887 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.318892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.318954 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:34:49.319092 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.319094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.319109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.319180 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.319364 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320570 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.321863 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.322501 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.322665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.322793 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.322832 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260620 18:34:49.323351 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.0.0.1:34114)
0620 18:34:49.319238 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.319248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.319289 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.319550 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.319551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.319553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.319593 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.319594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320789 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.320790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.322131 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.322134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.323138 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.323139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.323194 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.323198 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.323242 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.323282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260620 18:34:49.323606 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.10.109.65:34355)
0620 18:34:49.319113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.319117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.319347 (+ 230us) server_negotiation.cc:407] Connection header received
0620 18:34:49.319402 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.319405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.319423 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.319491 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.319665 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320709 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322223 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.322966 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.323036 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.323111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.323462 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.323503 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260620 18:34:49.323648 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 1a67280815804b17ae8de9f688517b4b (DEFAULT_TABLE table=test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.323964 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.10.109.67:37807)
0620 18:34:49.319063 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.319067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.319140 (+ 73us) server_negotiation.cc:407] Connection header received
0620 18:34:49.319218 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.319220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.319241 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.319354 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.319549 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320360 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322016 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.322703 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322855 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.322928 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.323797 (+ 869us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.323832 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260620 18:34:49.324060 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a67280815804b17ae8de9f688517b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.324150 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.318660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:37807 (local address 127.0.0.1:40738)
0620 18:34:49.319077 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.319088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.319154 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.319417 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.319418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.319421 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.319472 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.319473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.320440 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.320441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.321939 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.321940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.322954 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.322956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.323532 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.323536 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.323603 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.324084 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":90,"threads_started":1}
I20260620 18:34:49.324256 18316 tablet_service.cc:1467] Processing CreateTablet for tablet 1a67280815804b17ae8de9f688517b4b (DEFAULT_TABLE table=test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.324586 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a67280815804b17ae8de9f688517b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.328230 18690 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Bootstrap starting.
I20260620 18:34:49.328233 18691 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Bootstrap starting.
I20260620 18:34:49.328890 18690 tablet_bootstrap.cc:654] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.328890 18691 tablet_bootstrap.cc:654] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.328821 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 1a67280815804b17ae8de9f688517b4b (DEFAULT_TABLE table=test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.329144 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a67280815804b17ae8de9f688517b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.329231 18690 log.cc:824] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:49.329231 18691 log.cc:824] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:49.330019 18690 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: No bootstrap required, opened a new log
I20260620 18:34:49.330094 18690 ts_tablet_manager.cc:1397] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:49.330179 18691 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: No bootstrap required, opened a new log
I20260620 18:34:49.330238 18691 ts_tablet_manager.cc:1397] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:34:49.332115 18690 raft_consensus.cc:348] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.332172 18691 raft_consensus.cc:348] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.332291 18690 raft_consensus.cc:374] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.332307 18691 raft_consensus.cc:374] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.332330 18690 raft_consensus.cc:729] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eacff391e944b995edbb285209392b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.332341 18691 raft_consensus.cc:729] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.332472 18690 consensus_queue.cc:260] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.332472 18691 consensus_queue.cc:260] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.332738 18691 ts_tablet_manager.cc:1428] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:34:49.332738 18690 ts_tablet_manager.cc:1428] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:34:49.332774 18522 heartbeater.cc:503] Master 127.10.109.126:42279 was elected leader, sending a full tablet report...
I20260620 18:34:49.332849 18382 heartbeater.cc:503] Master 127.10.109.126:42279 was elected leader, sending a full tablet report...
I20260620 18:34:49.332928 18694 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Bootstrap starting.
I20260620 18:34:49.333556 18694 tablet_bootstrap.cc:654] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.333923 18694 log.cc:824] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:49.334728 18694 tablet_bootstrap.cc:492] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: No bootstrap required, opened a new log
I20260620 18:34:49.334926 18694 ts_tablet_manager.cc:1397] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:34:49.336254 18694 raft_consensus.cc:348] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.336396 18694 raft_consensus.cc:374] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.336440 18694 raft_consensus.cc:729] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97624f015c84c859ed28f62c45369fa, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.336552 18694 consensus_queue.cc:260] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.336807 18694 ts_tablet_manager.cc:1428] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:34:49.336886 18662 heartbeater.cc:503] Master 127.10.109.126:42279 was elected leader, sending a full tablet report...
I20260620 18:34:49.349747 18696 raft_consensus.cc:482] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:49.349877 18696 raft_consensus.cc:504] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.350227 18696 leader_election.cc:288] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807), 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355)
W20260620 18:34:49.351136 18523 tablet.cc:2369] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:49.353245 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.350451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:43599 (local address 127.10.109.67:37807)
0620 18:34:49.350490 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.350492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.350586 (+ 94us) server_negotiation.cc:407] Connection header received
0620 18:34:49.350617 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.350618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.350629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.350681 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.350903 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.351490 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.352456 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.352871 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.352925 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.352931 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.353123 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.353161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:49.353260 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.350420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:37807 (local address 127.10.109.66:43599)
0620 18:34:49.350473 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.350485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.350546 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.350715 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.350716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.350718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.350808 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.350810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.351571 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.351572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.352371 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.352374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.352951 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.352951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.353053 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.353057 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.353109 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.353151 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:34:49.353364 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.350563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:58821 (local address 127.10.109.65:34355)
0620 18:34:49.350602 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.350604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.350642 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:34:49.350781 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.350783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.350792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.350843 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.351010 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.351646 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.352586 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.352992 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.353060 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.353072 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.353248 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.353291 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:49.353536 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.350451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.10.109.66:58821)
0620 18:34:49.350582 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.350593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.350656 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.350870 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.350871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.350873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.350943 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.350945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.351725 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.351726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.352505 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.352507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.353070 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.353071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.353167 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.353171 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.353229 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.353268 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 18:34:49.353564 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a67280815804b17ae8de9f688517b4b" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa" is_pre_election: true
I20260620 18:34:49.353724 18616 raft_consensus.cc:2454] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 0.
I20260620 18:34:49.353957 18409 leader_election.cc:302] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eacff391e944b995edbb285209392b, c97624f015c84c859ed28f62c45369fa; no voters:
I20260620 18:34:49.353940 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a67280815804b17ae8de9f688517b4b" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5640511b999842d2891b6af2b5f7a39b" is_pre_election: true
I20260620 18:34:49.354086 18336 raft_consensus.cc:2454] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 0.
I20260620 18:34:49.354095 18696 raft_consensus.cc:2781] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:49.354144 18696 raft_consensus.cc:482] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:49.354175 18696 raft_consensus.cc:3037] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.355145 18696 raft_consensus.cc:504] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.355273 18696 leader_election.cc:288] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 election: Requested vote from peers c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807), 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355)
I20260620 18:34:49.355425 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a67280815804b17ae8de9f688517b4b" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa"
I20260620 18:34:49.355443 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a67280815804b17ae8de9f688517b4b" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5640511b999842d2891b6af2b5f7a39b"
I20260620 18:34:49.355496 18616 raft_consensus.cc:3037] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.355500 18336 raft_consensus.cc:3037] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.356287 18336 raft_consensus.cc:2454] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 1.
I20260620 18:34:49.356287 18616 raft_consensus.cc:2454] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 1.
I20260620 18:34:49.356463 18408 leader_election.cc:302] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.356562 18696 raft_consensus.cc:2781] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:49.356686 18696 raft_consensus.cc:686] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 1 LEADER]: Becoming Leader. State: Replica: 82eacff391e944b995edbb285209392b, State: Running, Role: LEADER
I20260620 18:34:49.356747 18696 consensus_queue.cc:237] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } }
I20260620 18:34:49.357412 18178 catalog_manager.cc:5495] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b reported cstate change: term changed from 0 to 1, leader changed from <none> to 82eacff391e944b995edbb285209392b (127.10.109.66). New cstate: current_term: 1 leader_uuid: "82eacff391e944b995edbb285209392b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:49.364945 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.361785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.10.109.66:35479)
0620 18:34:49.361867 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.361869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.361878 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:49.361908 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.361909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.361924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.361993 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:49.362203 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.362864 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.364054 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.364300 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.364381 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.364589 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.364709 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.364793 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.364841 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.364865 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:49.366811 18616 raft_consensus.cc:1261] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Refusing update from remote peer 82eacff391e944b995edbb285209392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.366809 18336 raft_consensus.cc:1261] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Refusing update from remote peer 82eacff391e944b995edbb285209392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.367113 18696 consensus_queue.cc:1035] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.367280 18696 consensus_queue.cc:1035] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.369449 18703 mvcc.cc:205] Tried to move back new op lower bound from 7298992084443688960 to 7298992084405784576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992084443688960}]
I20260620 18:34:49.369469 18704 mvcc.cc:205] Tried to move back new op lower bound from 7298992084443688960 to 7298992084405784576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992084443688960}]
I20260620 18:34:49.371670 18178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.371763 18178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.372478 18178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Sending DeleteTablet for 3 replicas of tablet 1a67280815804b17ae8de9f688517b4b
I20260620 18:34:49.372758 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a67280815804b17ae8de9f688517b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36210
I20260620 18:34:49.372766 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a67280815804b17ae8de9f688517b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:40738
I20260620 18:34:49.372845 18316 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a67280815804b17ae8de9f688517b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260620 18:34:49.373027 18714 tablet_replica.cc:330] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: stopping tablet replica
I20260620 18:34:49.373052 18712 tablet_replica.cc:330] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: stopping tablet replica
I20260620 18:34:49.373155 18714 raft_consensus.cc:2229] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.373253 18712 raft_consensus.cc:2229] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:49.373229 18178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:49.373355 18178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:49.373315 18714 raft_consensus.cc:2258] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.373682 18712 raft_consensus.cc:2258] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.373855 18713 tablet_replica.cc:330] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: stopping tablet replica
I20260620 18:34:49.373947 18713 raft_consensus.cc:2229] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.374059 18713 raft_consensus.cc:2258] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.374253 18714 ts_tablet_manager.cc:1905] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.374511 18712 ts_tablet_manager.cc:1905] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.374984 18316 tablet_service.cc:1467] Processing CreateTablet for tablet ce9e4dac6448438ebbd4906873f87075 (DEFAULT_TABLE table=test-table [id=20d13c1c787941a8b35cfb53f2fe713e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.374984 18596 tablet_service.cc:1467] Processing CreateTablet for tablet ce9e4dac6448438ebbd4906873f87075 (DEFAULT_TABLE table=test-table [id=20d13c1c787941a8b35cfb53f2fe713e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.375094 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9e4dac6448438ebbd4906873f87075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.375094 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9e4dac6448438ebbd4906873f87075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.376307 18714 ts_tablet_manager.cc:1918] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.376394 18694 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Bootstrap starting.
I20260620 18:34:49.376394 18691 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Bootstrap starting.
I20260620 18:34:49.376508 18714 log.cc:1196] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/1a67280815804b17ae8de9f688517b4b
I20260620 18:34:49.376804 18691 tablet_bootstrap.cc:654] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.376820 18694 tablet_bootstrap.cc:654] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.377233 18456 tablet_service.cc:1467] Processing CreateTablet for tablet ce9e4dac6448438ebbd4906873f87075 (DEFAULT_TABLE table=test-table [id=20d13c1c787941a8b35cfb53f2fe713e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.377343 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9e4dac6448438ebbd4906873f87075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.377417 18713 ts_tablet_manager.cc:1905] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.378661 18690 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Bootstrap starting.
I20260620 18:34:49.378808 18713 ts_tablet_manager.cc:1918] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.378882 18713 log.cc:1196] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/1a67280815804b17ae8de9f688517b4b
I20260620 18:34:49.379120 18690 tablet_bootstrap.cc:654] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.379156 18694 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: No bootstrap required, opened a new log
I20260620 18:34:49.379221 18694 ts_tablet_manager.cc:1397] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 18:34:49.378651 18712 ts_tablet_manager.cc:1918] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.379416 18694 raft_consensus.cc:348] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.379470 18712 log.cc:1196] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/1a67280815804b17ae8de9f688517b4b
I20260620 18:34:49.379493 18694 raft_consensus.cc:374] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.379525 18694 raft_consensus.cc:729] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97624f015c84c859ed28f62c45369fa, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.379580 18694 consensus_queue.cc:260] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.379685 18694 ts_tablet_manager.cc:1428] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.379717 18712 ts_tablet_manager.cc:1939] T 1a67280815804b17ae8de9f688517b4b P 82eacff391e944b995edbb285209392b: Deleting consensus metadata
I20260620 18:34:49.379837 18690 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: No bootstrap required, opened a new log
I20260620 18:34:49.379901 18690 ts_tablet_manager.cc:1397] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.380048 18690 raft_consensus.cc:348] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.380084 18163 catalog_manager.cc:4841] TS 82eacff391e944b995edbb285209392b (127.10.109.66:35479): tablet 1a67280815804b17ae8de9f688517b4b (table test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]) successfully deleted
I20260620 18:34:49.380131 18690 raft_consensus.cc:374] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.380163 18690 raft_consensus.cc:729] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eacff391e944b995edbb285209392b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.380198 18714 ts_tablet_manager.cc:1939] T 1a67280815804b17ae8de9f688517b4b P 5640511b999842d2891b6af2b5f7a39b: Deleting consensus metadata
I20260620 18:34:49.380211 18690 consensus_queue.cc:260] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.380318 18690 ts_tablet_manager.cc:1428] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:49.379139 18713 ts_tablet_manager.cc:1939] T 1a67280815804b17ae8de9f688517b4b P c97624f015c84c859ed28f62c45369fa: Deleting consensus metadata
I20260620 18:34:49.380605 18164 catalog_manager.cc:4841] TS 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355): tablet 1a67280815804b17ae8de9f688517b4b (table test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]) successfully deleted
I20260620 18:34:49.380913 18165 catalog_manager.cc:4841] TS c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807): tablet 1a67280815804b17ae8de9f688517b4b (table test-table [id=f900b2f2d12d4f55b5929bbe495a0bef]) successfully deleted
I20260620 18:34:49.381364 18691 tablet_bootstrap.cc:492] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: No bootstrap required, opened a new log
I20260620 18:34:49.381419 18691 ts_tablet_manager.cc:1397] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 18:34:49.381548 18691 raft_consensus.cc:348] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.381624 18691 raft_consensus.cc:374] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.381646 18691 raft_consensus.cc:729] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.381688 18691 consensus_queue.cc:260] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.381781 18691 ts_tablet_manager.cc:1428] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 18:34:49.443890 18383 tablet.cc:2369] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:49.453015 18696 raft_consensus.cc:482] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:49.453131 18696 raft_consensus.cc:504] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.453337 18696 leader_election.cc:288] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.453539 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9e4dac6448438ebbd4906873f87075" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5640511b999842d2891b6af2b5f7a39b" is_pre_election: true
I20260620 18:34:49.453529 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9e4dac6448438ebbd4906873f87075" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa" is_pre_election: true
I20260620 18:34:49.453655 18336 raft_consensus.cc:2454] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 0.
I20260620 18:34:49.453655 18616 raft_consensus.cc:2454] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 0.
I20260620 18:34:49.453804 18408 leader_election.cc:302] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.453948 18696 raft_consensus.cc:2781] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:49.453994 18696 raft_consensus.cc:482] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:49.454013 18696 raft_consensus.cc:3037] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.454792 18696 raft_consensus.cc:504] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.454941 18696 leader_election.cc:288] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 election: Requested vote from peers 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.455065 18336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9e4dac6448438ebbd4906873f87075" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5640511b999842d2891b6af2b5f7a39b"
I20260620 18:34:49.455092 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9e4dac6448438ebbd4906873f87075" candidate_uuid: "82eacff391e944b995edbb285209392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa"
I20260620 18:34:49.455139 18336 raft_consensus.cc:3037] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.455148 18616 raft_consensus.cc:3037] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.455888 18616 raft_consensus.cc:2454] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 1.
I20260620 18:34:49.455902 18336 raft_consensus.cc:2454] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82eacff391e944b995edbb285209392b in term 1.
I20260620 18:34:49.456061 18409 leader_election.cc:302] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82eacff391e944b995edbb285209392b, c97624f015c84c859ed28f62c45369fa; no voters:
I20260620 18:34:49.456152 18696 raft_consensus.cc:2781] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:49.456199 18696 raft_consensus.cc:686] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 1 LEADER]: Becoming Leader. State: Replica: 82eacff391e944b995edbb285209392b, State: Running, Role: LEADER
I20260620 18:34:49.456255 18696 consensus_queue.cc:237] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.456872 18178 catalog_manager.cc:5495] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b reported cstate change: term changed from 0 to 1, leader changed from <none> to 82eacff391e944b995edbb285209392b (127.10.109.66). New cstate: current_term: 1 leader_uuid: "82eacff391e944b995edbb285209392b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980488, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.126@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:49.491340 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.483413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:42279 (local address 127.0.0.1:45056)
0620 18:34:49.483563 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.483574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.483645 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.483788 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.483788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.484140 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:49.484195 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.484197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.484860 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.484861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.485736 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.485738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.486443 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.486444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.486502 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.487108 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:49.487113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:49.488697 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:49.490238 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.490243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.490245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.490433 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.490660 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:49.490661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:49.490661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:49.490723 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:49.491076 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:49.491081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:49.491122 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.491178 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.491267 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260620 18:34:49.491647 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.483467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.10.109.126:42279)
0620 18:34:49.483535 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.483537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.483639 (+ 102us) server_negotiation.cc:407] Connection header received
0620 18:34:49.483702 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.483703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.483711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.483760 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.484269 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.484795 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.485960 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.486255 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.486426 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.488785 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.488786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.488787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.488794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.490132 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.490505 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.490506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.490507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.490563 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.490872 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.490873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.490873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.491006 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.491077 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.491227 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.491336 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":44}
I20260620 18:34:49.493585 18178 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:45056:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:34:49.493757 18178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:49.495147 18456 tablet_service.cc:1467] Processing CreateTablet for tablet adab44bd27864d0ba95f493f5398bc96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e4e0d54b5f4f74b0fc91e972060026]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:49.495141 18316 tablet_service.cc:1467] Processing CreateTablet for tablet adab44bd27864d0ba95f493f5398bc96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e4e0d54b5f4f74b0fc91e972060026]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:49.495296 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab44bd27864d0ba95f493f5398bc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.495311 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab44bd27864d0ba95f493f5398bc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.496770 18690 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: Bootstrap starting.
I20260620 18:34:49.496876 18596 tablet_service.cc:1467] Processing CreateTablet for tablet adab44bd27864d0ba95f493f5398bc96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e4e0d54b5f4f74b0fc91e972060026]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:34:49.497040 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adab44bd27864d0ba95f493f5398bc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.497115 18690 tablet_bootstrap.cc:654] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.497380 18691 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: Bootstrap starting.
I20260620 18:34:49.497702 18690 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: No bootstrap required, opened a new log
I20260620 18:34:49.497754 18690 ts_tablet_manager.cc:1397] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.497820 18691 tablet_bootstrap.cc:654] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.497915 18690 raft_consensus.cc:348] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.497994 18690 raft_consensus.cc:374] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.498016 18690 raft_consensus.cc:729] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eacff391e944b995edbb285209392b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.498067 18690 consensus_queue.cc:260] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.498162 18690 ts_tablet_manager.cc:1428] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.498176 18696 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.498245 18696 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.498382 18694 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: Bootstrap starting.
I20260620 18:34:49.498804 18694 tablet_bootstrap.cc:654] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.498811 18691 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: No bootstrap required, opened a new log
I20260620 18:34:49.498849 18691 ts_tablet_manager.cc:1397] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.498993 18691 raft_consensus.cc:348] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.499071 18691 raft_consensus.cc:374] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.499110 18691 raft_consensus.cc:729] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.499164 18691 consensus_queue.cc:260] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.499286 18691 ts_tablet_manager.cc:1428] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.499338 18695 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.499415 18694 tablet_bootstrap.cc:492] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: No bootstrap required, opened a new log
I20260620 18:34:49.499456 18695 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.499464 18694 ts_tablet_manager.cc:1397] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:49.499625 18694 raft_consensus.cc:348] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.499675 18694 raft_consensus.cc:374] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.499688 18694 raft_consensus.cc:729] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97624f015c84c859ed28f62c45369fa, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.499754 18694 consensus_queue.cc:260] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.499879 18694 ts_tablet_manager.cc:1428] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.499961 18698 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.500079 18698 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 18:34:49.520838 18663 tablet.cc:2369] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:49.833465 18695 raft_consensus.cc:482] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:49.833556 18695 raft_consensus.cc:504] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.834263 18695 leader_election.cc:288] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.837304 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.834410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35479 (local address 127.10.109.65:35215)
0620 18:34:49.834574 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.834601 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.834699 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.834965 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.834967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.834969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.835053 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.835055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.835702 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.835704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.836489 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.836491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.837032 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.837033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.837105 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.837109 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.837176 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.837216 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260620 18:34:49.837291 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.834558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:35215 (local address 127.10.109.66:35479)
0620 18:34:49.834610 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.834612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.834664 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:34:49.834823 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.834825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.834872 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.834955 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.835143 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.835616 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.836583 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.836947 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.837032 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.837042 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.837181 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.837216 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:34:49.837548 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.834574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:33229 (local address 127.10.109.67:37807)
0620 18:34:49.834665 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.834668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.834881 (+ 213us) server_negotiation.cc:407] Connection header received
0620 18:34:49.835005 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.835007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.835019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.835282 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:49.835450 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.835928 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.836842 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.837186 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.837278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.837285 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.837433 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.837479 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:49.837545 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab44bd27864d0ba95f493f5398bc96" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b" is_pre_election: true
I20260620 18:34:49.837692 18476 raft_consensus.cc:2454] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.837764 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.834421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:37807 (local address 127.10.109.65:33229)
0620 18:34:49.834697 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.834710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.834904 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.835273 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.835275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.835278 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:49.835342 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.835344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.836020 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.836021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.836747 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.836749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.837278 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.837279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.837338 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.837341 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.837637 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.837677 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":63,"threads_started":1}
I20260620 18:34:49.837918 18267 leader_election.cc:302] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.838064 18695 raft_consensus.cc:2781] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:49.838110 18695 raft_consensus.cc:482] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:49.838143 18695 raft_consensus.cc:3037] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.838145 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab44bd27864d0ba95f493f5398bc96" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa" is_pre_election: true
I20260620 18:34:49.838230 18616 raft_consensus.cc:2454] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.838943 18695 raft_consensus.cc:504] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.839085 18695 leader_election.cc:288] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Requested vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.839246 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab44bd27864d0ba95f493f5398bc96" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b"
I20260620 18:34:49.839282 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adab44bd27864d0ba95f493f5398bc96" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa"
I20260620 18:34:49.839341 18616 raft_consensus.cc:3037] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.839341 18476 raft_consensus.cc:3037] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.839949 18616 raft_consensus.cc:2454] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.840133 18269 leader_election.cc:302] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, c97624f015c84c859ed28f62c45369fa; no voters:
I20260620 18:34:49.840147 18476 raft_consensus.cc:2454] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.840267 18695 raft_consensus.cc:2781] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:49.840420 18695 raft_consensus.cc:686] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [term 1 LEADER]: Becoming Leader. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Running, Role: LEADER
I20260620 18:34:49.840507 18695 consensus_queue.cc:237] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.840940 18730 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: TxnStatusTablet state changed. Reason: New leader 5640511b999842d2891b6af2b5f7a39b. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.841041 18730 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:49.841210 18732 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:34:49.841311 18732 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:34:49.841336 18178 catalog_manager.cc:5495] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5640511b999842d2891b6af2b5f7a39b (127.10.109.65). New cstate: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:49.853704 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.851086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.10.109.65:34355)
0620 18:34:49.851150 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.851152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.851160 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:49.851234 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.851235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.851252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.851316 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:49.851442 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.852076 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.852887 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.853091 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.853170 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.853365 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.853471 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.853545 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.853637 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.853655 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:34:49.853773 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.851022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.0.0.1:34116)
0620 18:34:49.851092 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.851102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.851153 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.851321 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.851322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.851323 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:49.851377 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.851380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.852200 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.852202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.852774 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.852776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.853167 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.853168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.853232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.853253 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.853555 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.853565 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.853632 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.853700 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:34:49.854833 18616 raft_consensus.cc:1261] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.854902 18476 raft_consensus.cc:1261] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.855170 18730 consensus_queue.cc:1035] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.855273 18695 consensus_queue.cc:1035] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.855820 18696 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: TxnStatusTablet state changed. Reason: New leader 5640511b999842d2891b6af2b5f7a39b. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.855917 18696 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.856237 18698 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: TxnStatusTablet state changed. Reason: New leader 5640511b999842d2891b6af2b5f7a39b. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.856336 18698 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.856895 18695 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.856956 18730 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.857148 18730 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:49.857196 18696 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.857285 18696 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 82eacff391e944b995edbb285209392b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.856978 18695 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P 5640511b999842d2891b6af2b5f7a39b: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:34:49.858754 18698 tablet_replica.cc:439] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } } }
I20260620 18:34:49.858834 18698 tablet_replica.cc:442] T adab44bd27864d0ba95f493f5398bc96 P c97624f015c84c859ed28f62c45369fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:34:49.865541 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.860281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:35213 (local address 127.10.109.65:34355)
0620 18:34:49.860322 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.860324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.862963 (+ 2639us) server_negotiation.cc:407] Connection header received
0620 18:34:49.863017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.863018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.863029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.863094 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:49.863169 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.863879 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.864640 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.864945 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.865013 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.865255 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.865286 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.865342 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.865455 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.865474 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:34:49.865564 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.860188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.10.109.66:35213)
0620 18:34:49.862883 (+ 2695us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.862895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.862956 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.863096 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.863096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.863097 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:49.863133 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.863134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.863954 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.863955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.864564 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.864565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.865054 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.865055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.865120 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.865135 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.865364 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.865365 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.865450 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.865500 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2665,"thread_start_us":44,"threads_started":1}
I20260620 18:34:49.867679 18616 raft_consensus.cc:1261] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Refusing update from remote peer 82eacff391e944b995edbb285209392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.867704 18336 raft_consensus.cc:1261] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Refusing update from remote peer 82eacff391e944b995edbb285209392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.867869 18700 consensus_queue.cc:1035] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.867947 18696 consensus_queue.cc:1035] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.869781 18705 mvcc.cc:205] Tried to move back new op lower bound from 7298992086496841728 to 7298992084813258752. Current Snapshot: MvccSnapshot[applied={T|T < 7298992086496841728}]
I20260620 18:34:49.876808 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.873762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:55059 (local address 127.10.109.66:35479)
0620 18:34:49.873802 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.873804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.873866 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:34:49.873961 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.873962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.873970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.874008 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:49.874131 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.874574 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.876115 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.876308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.876371 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.876564 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.876591 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.876633 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.876705 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.876722 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:49.877048 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.873593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35479 (local address 127.10.109.65:55059)
0620 18:34:49.873799 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:49.873809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:49.873892 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:49.874011 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:49.874011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:49.874013 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:49.874078 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.874080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.874662 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.874663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.875998 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:49.876000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.876407 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.876409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:49.876473 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.876487 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.876648 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.876650 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:49.876702 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:34:49.876743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260620 18:34:49.885128 18177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.885231 18177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.885838 18177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Sending DeleteTablet for 3 replicas of tablet ce9e4dac6448438ebbd4906873f87075
I20260620 18:34:49.886097 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9e4dac6448438ebbd4906873f87075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:40738
I20260620 18:34:49.886346 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9e4dac6448438ebbd4906873f87075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36210
I20260620 18:34:49.886533 18316 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9e4dac6448438ebbd4906873f87075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260620 18:34:49.886662 18747 tablet_replica.cc:330] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: stopping tablet replica
I20260620 18:34:49.886711 18747 raft_consensus.cc:2229] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.886770 18747 raft_consensus.cc:2258] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.886998 18747 ts_tablet_manager.cc:1905] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.888860 18749 tablet_replica.cc:330] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: stopping tablet replica
I20260620 18:34:49.888929 18749 raft_consensus.cc:2229] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.888953 18747 ts_tablet_manager.cc:1918] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:49.888970 18749 raft_consensus.cc:2258] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.889005 18747 log.cc:1196] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/ce9e4dac6448438ebbd4906873f87075
I20260620 18:34:49.889230 18747 ts_tablet_manager.cc:1939] T ce9e4dac6448438ebbd4906873f87075 P c97624f015c84c859ed28f62c45369fa: Deleting consensus metadata
I20260620 18:34:49.889222 18749 ts_tablet_manager.cc:1905] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.889339 18748 tablet_replica.cc:330] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: stopping tablet replica
I20260620 18:34:49.889606 18165 catalog_manager.cc:4841] TS c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807): tablet ce9e4dac6448438ebbd4906873f87075 (table test-table [id=20d13c1c787941a8b35cfb53f2fe713e]) successfully deleted
I20260620 18:34:49.889740 18748 raft_consensus.cc:2229] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:49.889861 18748 raft_consensus.cc:2258] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.890152 18748 ts_tablet_manager.cc:1905] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.890951 18749 ts_tablet_manager.cc:1918] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:49.891013 18749 log.cc:1196] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/ce9e4dac6448438ebbd4906873f87075
I20260620 18:34:49.891243 18749 ts_tablet_manager.cc:1939] T ce9e4dac6448438ebbd4906873f87075 P 5640511b999842d2891b6af2b5f7a39b: Deleting consensus metadata
I20260620 18:34:49.891590 18164 catalog_manager.cc:4841] TS 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355): tablet ce9e4dac6448438ebbd4906873f87075 (table test-table [id=20d13c1c787941a8b35cfb53f2fe713e]) successfully deleted
I20260620 18:34:49.892122 18748 ts_tablet_manager.cc:1918] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:49.892230 18748 log.cc:1196] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/ce9e4dac6448438ebbd4906873f87075
I20260620 18:34:49.892503 18748 ts_tablet_manager.cc:1939] T ce9e4dac6448438ebbd4906873f87075 P 82eacff391e944b995edbb285209392b: Deleting consensus metadata
I20260620 18:34:49.892819 18163 catalog_manager.cc:4841] TS 82eacff391e944b995edbb285209392b (127.10.109.66:35479): tablet ce9e4dac6448438ebbd4906873f87075 (table test-table [id=20d13c1c787941a8b35cfb53f2fe713e]) successfully deleted
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:34:49.913712 18228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:49.907230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.10.109.126:42279)
0620 18:34:49.907298 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.907300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.907449 (+ 149us) server_negotiation.cc:407] Connection header received
0620 18:34:49.907503 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.907504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.907515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.907598 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.908156 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.908963 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.909741 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.910002 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.910081 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.913635 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45058: BlockingRecv error: recv got EOF from 127.0.0.1:45058 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 18:34:49 dist-test-slave-3jql krb5kdc[18134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM
I20260620 18:34:49.923321 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.914854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.10.109.126:42279)
0620 18:34:49.914896 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.914898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.915020 (+ 122us) server_negotiation.cc:407] Connection header received
0620 18:34:49.915075 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.915076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.915084 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.915145 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:49.915719 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.916193 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.917159 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.917334 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.917404 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.920000 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:49.920003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:49.920004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:49.920014 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:49.921752 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.922378 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.922379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.922379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.922451 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:49.922833 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:49.922833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:49.922834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:49.922973 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:49.923053 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.923196 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.923257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:49.925530 18177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:49.925714 18177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:49.927168 18316 tablet_service.cc:1467] Processing CreateTablet for tablet 3c84558a27c34993bd4f1fe177b38deb (DEFAULT_TABLE table=test-table [id=82f3e7a1dbe249549fe134106453f14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.927309 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c84558a27c34993bd4f1fe177b38deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.927275 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 3c84558a27c34993bd4f1fe177b38deb (DEFAULT_TABLE table=test-table [id=82f3e7a1dbe249549fe134106453f14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.927393 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c84558a27c34993bd4f1fe177b38deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.928642 18694 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Bootstrap starting.
I20260620 18:34:49.928656 18691 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Bootstrap starting.
I20260620 18:34:49.928674 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 3c84558a27c34993bd4f1fe177b38deb (DEFAULT_TABLE table=test-table [id=82f3e7a1dbe249549fe134106453f14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.928788 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c84558a27c34993bd4f1fe177b38deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.929031 18691 tablet_bootstrap.cc:654] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.929042 18694 tablet_bootstrap.cc:654] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.929759 18691 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: No bootstrap required, opened a new log
I20260620 18:34:49.929811 18691 ts_tablet_manager.cc:1397] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.929951 18691 raft_consensus.cc:348] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.930043 18691 raft_consensus.cc:374] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.930065 18691 raft_consensus.cc:729] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.930109 18691 consensus_queue.cc:260] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.930233 18691 ts_tablet_manager.cc:1428] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:49.930240 18690 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Bootstrap starting.
I20260620 18:34:49.930585 18690 tablet_bootstrap.cc:654] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.931295 18694 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: No bootstrap required, opened a new log
I20260620 18:34:49.931324 18690 tablet_bootstrap.cc:492] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: No bootstrap required, opened a new log
I20260620 18:34:49.931367 18690 ts_tablet_manager.cc:1397] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.931366 18694 ts_tablet_manager.cc:1397] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 18:34:49.931470 18690 raft_consensus.cc:348] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.931541 18690 raft_consensus.cc:374] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.931563 18690 raft_consensus.cc:729] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eacff391e944b995edbb285209392b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.931526 18694 raft_consensus.cc:348] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.931608 18694 raft_consensus.cc:374] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.931607 18690 consensus_queue.cc:260] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.931632 18694 raft_consensus.cc:729] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97624f015c84c859ed28f62c45369fa, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.931689 18690 ts_tablet_manager.cc:1428] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.931676 18694 consensus_queue.cc:260] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.931764 18694 ts_tablet_manager.cc:1428] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:49.956598 18730 raft_consensus.cc:482] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:49.956707 18730 raft_consensus.cc:504] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.956897 18730 leader_election.cc:288] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.957103 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c84558a27c34993bd4f1fe177b38deb" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b" is_pre_election: true
I20260620 18:34:49.957108 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c84558a27c34993bd4f1fe177b38deb" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa" is_pre_election: true
I20260620 18:34:49.957197 18476 raft_consensus.cc:2454] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.957223 18616 raft_consensus.cc:2454] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.957365 18267 leader_election.cc:302] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.957448 18730 raft_consensus.cc:2781] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:49.957486 18730 raft_consensus.cc:482] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:49.957508 18730 raft_consensus.cc:3037] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.958173 18730 raft_consensus.cc:504] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.958323 18730 leader_election.cc:288] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Requested vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.958447 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c84558a27c34993bd4f1fe177b38deb" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b"
I20260620 18:34:49.958528 18476 raft_consensus.cc:3037] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.958515 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c84558a27c34993bd4f1fe177b38deb" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa"
I20260620 18:34:49.958583 18616 raft_consensus.cc:3037] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.959311 18616 raft_consensus.cc:2454] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.959311 18476 raft_consensus.cc:2454] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.959494 18267 leader_election.cc:302] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.959585 18730 raft_consensus.cc:2781] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:49.959633 18730 raft_consensus.cc:686] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 1 LEADER]: Becoming Leader. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Running, Role: LEADER
I20260620 18:34:49.959684 18730 consensus_queue.cc:237] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.960271 18177 catalog_manager.cc:5495] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5640511b999842d2891b6af2b5f7a39b (127.10.109.65). New cstate: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:49.972268 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:49.969530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.10.109.65:34355)
0620 18:34:49.969570 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:49.969573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:49.969582 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:49.969633 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:49.969633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:49.969641 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:49.969689 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:49.969808 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.970399 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.971483 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:49.971669 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:49.971730 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:49.971926 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:49.971957 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:49.972027 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:49.972110 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 18:34:49.972148 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:34:49.973124 18476 raft_consensus.cc:1261] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.973193 18616 raft_consensus.cc:1261] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:49.973318 18730 consensus_queue.cc:1035] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.973449 18737 consensus_queue.cc:1035] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:49.976923 18177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.977015 18177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:49.977595 18177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Sending DeleteTablet for 3 replicas of tablet 3c84558a27c34993bd4f1fe177b38deb
I20260620 18:34:49.977849 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c84558a27c34993bd4f1fe177b38deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36210
I20260620 18:34:49.977946 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c84558a27c34993bd4f1fe177b38deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:40738
I20260620 18:34:49.977972 18316 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c84558a27c34993bd4f1fe177b38deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:49 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260620 18:34:49.978029 18748 tablet_replica.cc:330] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: stopping tablet replica
I20260620 18:34:49.978040 18749 tablet_replica.cc:330] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: stopping tablet replica
I20260620 18:34:49.978076 18749 raft_consensus.cc:2229] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:49.978077 18748 raft_consensus.cc:2229] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.978071 18747 tablet_replica.cc:330] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: stopping tablet replica
I20260620 18:34:49.978116 18748 raft_consensus.cc:2258] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.978130 18747 raft_consensus.cc:2229] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:49.978149 18749 raft_consensus.cc:2258] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.978181 18747 raft_consensus.cc:2258] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:49.978348 18177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:49.978477 18177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:49.978716 18748 ts_tablet_manager.cc:1905] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.979063 18749 ts_tablet_manager.cc:1905] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.979895 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c9674113b4e349cd20150ba59f622 (DEFAULT_TABLE table=test-table [id=98bc3c66d7f84c91bbad0f2182fa3e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.980000 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c9674113b4e349cd20150ba59f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.980659 18749 ts_tablet_manager.cc:1918] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.980716 18749 log.cc:1196] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/3c84558a27c34993bd4f1fe177b38deb
I20260620 18:34:49.980882 18748 ts_tablet_manager.cc:1918] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.980929 18748 log.cc:1196] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/3c84558a27c34993bd4f1fe177b38deb
I20260620 18:34:49.980952 18749 ts_tablet_manager.cc:1939] T 3c84558a27c34993bd4f1fe177b38deb P 5640511b999842d2891b6af2b5f7a39b: Deleting consensus metadata
I20260620 18:34:49.981139 18748 ts_tablet_manager.cc:1939] T 3c84558a27c34993bd4f1fe177b38deb P 82eacff391e944b995edbb285209392b: Deleting consensus metadata
I20260620 18:34:49.981242 18690 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Bootstrap starting.
I20260620 18:34:49.981446 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c9674113b4e349cd20150ba59f622 (DEFAULT_TABLE table=test-table [id=98bc3c66d7f84c91bbad0f2182fa3e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.981554 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c9674113b4e349cd20150ba59f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.981578 18690 tablet_bootstrap.cc:654] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.981971 18747 ts_tablet_manager.cc:1905] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:49.982069 18690 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: No bootstrap required, opened a new log
I20260620 18:34:49.982107 18690 ts_tablet_manager.cc:1397] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.982196 18690 raft_consensus.cc:348] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.982262 18690 raft_consensus.cc:374] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.982286 18690 raft_consensus.cc:729] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82eacff391e944b995edbb285209392b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.982327 18690 consensus_queue.cc:260] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.982421 18690 ts_tablet_manager.cc:1428] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.982992 18694 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Bootstrap starting.
I20260620 18:34:49.983453 18747 ts_tablet_manager.cc:1918] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:49.983515 18747 log.cc:1196] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/3c84558a27c34993bd4f1fe177b38deb
I20260620 18:34:49.983570 18694 tablet_bootstrap.cc:654] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.983724 18747 ts_tablet_manager.cc:1939] T 3c84558a27c34993bd4f1fe177b38deb P c97624f015c84c859ed28f62c45369fa: Deleting consensus metadata
I20260620 18:34:49.983968 18163 catalog_manager.cc:4841] TS 82eacff391e944b995edbb285209392b (127.10.109.66:35479): tablet 3c84558a27c34993bd4f1fe177b38deb (table test-table [id=82f3e7a1dbe249549fe134106453f14c]) successfully deleted
I20260620 18:34:49.984023 18164 catalog_manager.cc:4841] TS 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355): tablet 3c84558a27c34993bd4f1fe177b38deb (table test-table [id=82f3e7a1dbe249549fe134106453f14c]) successfully deleted
I20260620 18:34:49.984180 18316 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c9674113b4e349cd20150ba59f622 (DEFAULT_TABLE table=test-table [id=98bc3c66d7f84c91bbad0f2182fa3e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:49.984254 18165 catalog_manager.cc:4841] TS c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807): tablet 3c84558a27c34993bd4f1fe177b38deb (table test-table [id=82f3e7a1dbe249549fe134106453f14c]) successfully deleted
I20260620 18:34:49.984279 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c9674113b4e349cd20150ba59f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:49.985138 18691 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Bootstrap starting.
I20260620 18:34:49.985477 18694 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: No bootstrap required, opened a new log
I20260620 18:34:49.985522 18694 ts_tablet_manager.cc:1397] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 18:34:49.985548 18691 tablet_bootstrap.cc:654] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:49.985668 18694 raft_consensus.cc:348] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.985742 18694 raft_consensus.cc:374] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.985764 18694 raft_consensus.cc:729] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97624f015c84c859ed28f62c45369fa, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.985816 18694 consensus_queue.cc:260] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.985905 18694 ts_tablet_manager.cc:1428] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 18:34:49.986198 18691 tablet_bootstrap.cc:492] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: No bootstrap required, opened a new log
I20260620 18:34:49.986243 18691 ts_tablet_manager.cc:1397] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:49.986364 18691 raft_consensus.cc:348] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.986441 18691 raft_consensus.cc:374] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:49.986459 18691 raft_consensus.cc:729] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Initialized, Role: FOLLOWER
I20260620 18:34:49.986497 18691 consensus_queue.cc:260] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.986564 18691 ts_tablet_manager.cc:1428] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:49.990267 18695 raft_consensus.cc:482] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:49.990386 18695 raft_consensus.cc:504] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.990550 18695 leader_election.cc:288] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.990962 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c9674113b4e349cd20150ba59f622" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b" is_pre_election: true
I20260620 18:34:49.991080 18476 raft_consensus.cc:2454] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.991220 18267 leader_election.cc:302] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, 82eacff391e944b995edbb285209392b; no voters:
I20260620 18:34:49.991226 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c9674113b4e349cd20150ba59f622" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa" is_pre_election: true
I20260620 18:34:49.991324 18616 raft_consensus.cc:2454] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 0.
I20260620 18:34:49.991370 18695 raft_consensus.cc:2781] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:49.991407 18695 raft_consensus.cc:482] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:49.991432 18695 raft_consensus.cc:3037] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.992180 18695 raft_consensus.cc:504] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.992314 18695 leader_election.cc:288] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Requested vote from peers 82eacff391e944b995edbb285209392b (127.10.109.66:35479), c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807)
I20260620 18:34:49.992583 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c9674113b4e349cd20150ba59f622" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97624f015c84c859ed28f62c45369fa"
I20260620 18:34:49.992662 18616 raft_consensus.cc:3037] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.992630 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c9674113b4e349cd20150ba59f622" candidate_uuid: "5640511b999842d2891b6af2b5f7a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82eacff391e944b995edbb285209392b"
I20260620 18:34:49.992694 18476 raft_consensus.cc:3037] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:49.993220 18616 raft_consensus.cc:2454] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.993363 18476 raft_consensus.cc:2454] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5640511b999842d2891b6af2b5f7a39b in term 1.
I20260620 18:34:49.993522 18269 leader_election.cc:302] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5640511b999842d2891b6af2b5f7a39b, c97624f015c84c859ed28f62c45369fa; no voters:
I20260620 18:34:49.993628 18695 raft_consensus.cc:2781] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:49.993685 18695 raft_consensus.cc:686] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 1 LEADER]: Becoming Leader. State: Replica: 5640511b999842d2891b6af2b5f7a39b, State: Running, Role: LEADER
I20260620 18:34:49.993728 18695 consensus_queue.cc:237] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } }
I20260620 18:34:49.994179 18177 catalog_manager.cc:5495] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5640511b999842d2891b6af2b5f7a39b (127.10.109.65). New cstate: current_term: 1 leader_uuid: "5640511b999842d2891b6af2b5f7a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5640511b999842d2891b6af2b5f7a39b" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 34355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20260620 18:34:50.004472 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:50.001584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.10.109.65:59203)
0620 18:34:50.001644 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:50.001656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:50.001696 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:50.001894 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:50.001894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:50.001896 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:34:50.001935 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:50.001936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.002930 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.002932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:50.003492 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:50.003494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.004035 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.004036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:50.004078 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:50.004084 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:34:50.004312 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:34:50.004314 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:34:50.004359 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 18:34:50.004398 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:34:50.004482 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:50.001650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:59203 (local address 127.10.109.65:34355)
0620 18:34:50.001686 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:50.001688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:50.001696 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:50.001795 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:50.001795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:50.001806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:50.001864 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:50.001995 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.002735 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.003612 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:50.003928 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:50.003994 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:50.004155 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:50.004185 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:50.004256 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:50.004386 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:34:50.004408 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:34:50.007117 18616 raft_consensus.cc:1261] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:50.007438 18476 raft_consensus.cc:1261] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Refusing update from remote peer 5640511b999842d2891b6af2b5f7a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:50.007733 18737 consensus_queue.cc:1035] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97624f015c84c859ed28f62c45369fa" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:50.007879 18695 consensus_queue.cc:1035] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82eacff391e944b995edbb285209392b" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:50.019981 18177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:50.020114 18177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:50.020690 18177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe53db8204874e57998c4fafe53056ef: Sending DeleteTablet for 3 replicas of tablet 5b7c9674113b4e349cd20150ba59f622
I20260620 18:34:50.020905 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c9674113b4e349cd20150ba59f622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:50 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36210
I20260620 18:34:50.021030 18748 tablet_replica.cc:330] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: stopping tablet replica
I20260620 18:34:50.021031 18316 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c9674113b4e349cd20150ba59f622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:50 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:34114
I20260620 18:34:50.021087 18748 raft_consensus.cc:2229] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:50.021116 18749 tablet_replica.cc:330] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: stopping tablet replica
I20260620 18:34:50.021127 18748 raft_consensus.cc:2258] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:50.021154 18749 raft_consensus.cc:2229] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:50.021214 18749 raft_consensus.cc:2258] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:50.021358 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c9674113b4e349cd20150ba59f622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:50 UTC) from {username='oryx', principal='oryx/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:40738
I20260620 18:34:50.021456 18747 tablet_replica.cc:330] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: stopping tablet replica
I20260620 18:34:50.021503 18747 raft_consensus.cc:2229] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:50.021543 18747 raft_consensus.cc:2258] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:50.021631 18749 ts_tablet_manager.cc:1905] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:50.021857 18748 ts_tablet_manager.cc:1905] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:50.021930 18747 ts_tablet_manager.cc:1905] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:50.023811 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18251
I20260620 18:34:50.023934 18749 ts_tablet_manager.cc:1918] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:50.023983 18749 log.cc:1196] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/5b7c9674113b4e349cd20150ba59f622
I20260620 18:34:50.024169 18749 ts_tablet_manager.cc:1939] T 5b7c9674113b4e349cd20150ba59f622 P 5640511b999842d2891b6af2b5f7a39b: Deleting consensus metadata
I20260620 18:34:50.025266 18747 ts_tablet_manager.cc:1918] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:50.025321 18747 log.cc:1196] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/5b7c9674113b4e349cd20150ba59f622
I20260620 18:34:50.025527 18747 ts_tablet_manager.cc:1939] T 5b7c9674113b4e349cd20150ba59f622 P c97624f015c84c859ed28f62c45369fa: Deleting consensus metadata
I20260620 18:34:50.025919 18165 catalog_manager.cc:4841] TS c97624f015c84c859ed28f62c45369fa (127.10.109.67:37807): tablet 5b7c9674113b4e349cd20150ba59f622 (table test-table [id=98bc3c66d7f84c91bbad0f2182fa3e50]) successfully deleted
I20260620 18:34:50.027086 18748 ts_tablet_manager.cc:1918] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:34:50.027180 18748 log.cc:1196] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/5b7c9674113b4e349cd20150ba59f622
I20260620 18:34:50.027442 18748 ts_tablet_manager.cc:1939] T 5b7c9674113b4e349cd20150ba59f622 P 82eacff391e944b995edbb285209392b: Deleting consensus metadata
I20260620 18:34:50.027963 18163 catalog_manager.cc:4841] TS 82eacff391e944b995edbb285209392b (127.10.109.66:35479): tablet 5b7c9674113b4e349cd20150ba59f622 (table test-table [id=98bc3c66d7f84c91bbad0f2182fa3e50]) successfully deleted
W20260620 18:34:50.030380 18164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:34355 (error 108)
I20260620 18:34:50.030696 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18391
I20260620 18:34:50.032298 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:50.030809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:34355 (local address 127.0.0.1:34128)
0620 18:34:50.032229 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:50.032266 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":38,"threads_started":1}
W20260620 18:34:50.032459 18164 catalog_manager.cc:4584] TS 5640511b999842d2891b6af2b5f7a39b (127.10.109.65:34355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b7c9674113b4e349cd20150ba59f622: Network error: Client connection negotiation failed: client connection to 127.10.109.65:34355: connect: Connection refused (error 111)
I20260620 18:34:50.037902 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18531
I20260620 18:34:50.043836 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18147
2026-06-20T18:34:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2590 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:50 dist-test-slave-3jql krb5kdc[18780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:50 dist-test-slave-3jql krb5kdc[18780](info): set up 2 sockets
Jun 20 18:34:50 dist-test-slave-3jql krb5kdc[18780](info): commencing operation
krb5kdc: starting...
W20260620 18:34:51.087582 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:51 dist-test-slave-3jql krb5kdc[18780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:51.206550 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34239
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:51.285439 18793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:51.285622 18793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:51.285645 18793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:51.286688 18793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:51.286722 18793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:51.286795 18793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:51.286824 18793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:51.286846 18793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:51.288239 18793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:34239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:51.288411 18793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:51.288561 18793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:51.293174 18798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:51.293219 18799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:51.293198 18801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:51.293403 18793 server_base.cc:1034] running on GCE node
I20260620 18:34:51.293651 18793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:51.293881 18793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:51.295024 18793 hybrid_clock.cc:648] HybridClock initialized: now 1781980491295009 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:34:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:51 dist-test-slave-3jql krb5kdc[18806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:51 dist-test-slave-3jql krb5kdc[18806](info): set up 2 sockets
Jun 20 18:34:51 dist-test-slave-3jql krb5kdc[18806](info): commencing operation
krb5kdc: starting...
W20260620 18:34:52.356241 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:34:52.482470 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36535
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:52.560896 18819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:52.561132 18819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:52.561168 18819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:52.562423 18819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:52.562490 18819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:52.562517 18819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:52.562548 18819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:52.562577 18819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:52.564020 18819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:52.564282 18819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:52.564503 18819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:52.569151 18824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.569166 18825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.569468 18827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:52.569622 18819 server_base.cc:1034] running on GCE node
I20260620 18:34:52.569921 18819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:52.570210 18819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:52.571398 18819 hybrid_clock.cc:648] HybridClock initialized: now 1781980492571386 us; error 50 us; skew 500 ppm
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:52.578986 18819 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:52.579408 18819 webserver.cc:460] Webserver started at http://127.10.109.126:42829/ using document root <none> and password file <none>
I20260620 18:34:52.579628 18819 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:52.579695 18819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:52.579876 18819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:52.580936 18819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "67e80c0629d8d4f70dc856c91822d77f"
server_key_iv: "54abddb5113f2c433b96f43ccd0abe4d"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.581293 18819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "67e80c0629d8d4f70dc856c91822d77f"
server_key_iv: "54abddb5113f2c433b96f43ccd0abe4d"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.582592 18819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:52.583455 18834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.583670 18819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:52.583771 18819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "67e80c0629d8d4f70dc856c91822d77f"
server_key_iv: "54abddb5113f2c433b96f43ccd0abe4d"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.583892 18819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:52.593532 18819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:52.595566 18819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:52.595734 18819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:52.600236 18819 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36535
I20260620 18:34:52.600267 18886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36535 every 8 connection(s)
I20260620 18:34:52.600644 18819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:52.601544 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:52.603780 18887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Bootstrap starting.
I20260620 18:34:52.604334 18887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:52.604607 18887 log.cc:824] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:52.605298 18887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: No bootstrap required, opened a new log
I20260620 18:34:52.606927 18887 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:52.607054 18887 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:52.607076 18887 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bcd83d12d844d5b1aaadab3e1a54f3, State: Initialized, Role: FOLLOWER
I20260620 18:34:52.607251 18887 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:52.607309 18887 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:52.607374 18887 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:52.607369 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18819
I20260620 18:34:52.607424 18887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:52.607497 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:34:52.607712 10677 external_mini_cluster.cc:1428] Setting key 4dc2262c03f2fedd27e27ce33208fd55
I20260620 18:34:52.608040 18887 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:52.608163 18887 leader_election.cc:302] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5bcd83d12d844d5b1aaadab3e1a54f3; no voters:
I20260620 18:34:52.608310 18887 leader_election.cc:288] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:52.608553 18887 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:52.608556 18893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:52.608886 18893 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 LEADER]: Becoming Leader. State: Replica: e5bcd83d12d844d5b1aaadab3e1a54f3, State: Running, Role: LEADER
I20260620 18:34:52.608990 18893 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:52.609313 18893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5bcd83d12d844d5b1aaadab3e1a54f3. Latest consensus state: current_term: 1 leader_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } } }
I20260620 18:34:52.609401 18893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:52.609606 18901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:52.610139 18901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:52.611439 18892 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } } }
I20260620 18:34:52.611522 18892 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:52.611773 18901 catalog_manager.cc:1347] Generated new cluster ID: 425425e58d564162a51998d199677d5b
I20260620 18:34:52.611828 18901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:52.620016 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:52.608195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.10.109.126:36535)
0620 18:34:52.608386 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:52.608395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:52.608434 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:34:52.608579 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:52.608586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:52.608646 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:52.608781 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:52.609632 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.611133 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.612214 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.612825 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.612945 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:52.615929 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:52.615933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:52.615933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:52.615945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:52.617968 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.618349 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.618352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.618352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.618466 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.618667 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.618668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.618668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.619055 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:52.619597 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:52.619667 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:34:52.619802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260620 18:34:52.620074 18901 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:52.620823 18901 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:52.631115 18901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Generated new TSK 0
I20260620 18:34:52.631291 18901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:34:52.657928 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:52.740985 18913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:52.741183 18913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:52.741207 18913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:52.742362 18913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:52.742401 18913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:52.742427 18913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:52.743811 18913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:52.743990 18913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:52.744155 18913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:52.749250 18918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.749222 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.749231 18921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:52.749456 18913 server_base.cc:1034] running on GCE node
I20260620 18:34:52.749972 18913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:52.750262 18913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:52.751439 18913 hybrid_clock.cc:648] HybridClock initialized: now 1781980492751424 us; error 55 us; skew 500 ppm
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:52.754058 18913 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:34:52.754474 18913 webserver.cc:460] Webserver started at http://127.10.109.65:40073/ using document root <none> and password file <none>
I20260620 18:34:52.754674 18913 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:52.754720 18913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:52.754900 18913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:52.755913 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "6523307100334832b7fa15fd316ff9ea"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "9f71ef8d7b83478b2e4c99718fbc4d02"
server_key_iv: "56db30c6b49976e0cbed9bf96e75a74c"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.756227 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "6523307100334832b7fa15fd316ff9ea"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "9f71ef8d7b83478b2e4c99718fbc4d02"
server_key_iv: "56db30c6b49976e0cbed9bf96e75a74c"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.757653 18913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:52.758466 18928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.758687 18913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:52.758809 18913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "6523307100334832b7fa15fd316ff9ea"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "9f71ef8d7b83478b2e4c99718fbc4d02"
server_key_iv: "56db30c6b49976e0cbed9bf96e75a74c"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.758873 18913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:52.782047 18913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:52.784526 18913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:52.784669 18913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:52.784873 18913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:52.785225 18913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:52.785292 18913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.785332 18913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:52.785383 18913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.792014 18913 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:42795
I20260620 18:34:52.792057 19041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:42795 every 8 connection(s)
I20260620 18:34:52.792456 18913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:52.792919 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 18913
I20260620 18:34:52.793037 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:52.793244 10677 external_mini_cluster.cc:1428] Setting key b55bc5a751a96da10466b35ba5966728
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
I20260620 18:34:52.803946 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:52.793338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:36931 (local address 127.10.109.126:36535)
0620 18:34:52.793405 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:52.793407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:52.793656 (+ 249us) server_negotiation.cc:407] Connection header received
0620 18:34:52.794311 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:52.794312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:52.794329 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:52.794394 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:52.795442 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.795957 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.797517 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.797804 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.797863 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:52.800490 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:52.800491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:52.800492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:52.800500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:52.802103 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.802866 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.802867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.802868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.802931 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.803229 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.803230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.803230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.803398 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:52.803498 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:52.803764 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:34:52.803846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:34:52.804090 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:52.793308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:36931)
0620 18:34:52.793495 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:52.793542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:52.794202 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:52.794469 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:52.794472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:52.795082 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:52.795324 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:52.795331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.796049 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.796050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:52.797395 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:52.797397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.797897 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.797898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:52.798048 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:52.798548 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:52.798560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:52.800356 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:52.802294 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:52.802311 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:52.802314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:52.802714 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:52.803005 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:52.803006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:52.803006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:52.803114 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:52.803495 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:52.803501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:52.803626 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 18:34:52.803755 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:34:52.803885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
I20260620 18:34:52.804646 19042 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36535
I20260620 18:34:52.804764 19042 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:52.804934 19042 heartbeater.cc:511] Master 127.10.109.126:36535 requested a full tablet report, sending...
I20260620 18:34:52.805478 18851 ts_manager.cc:194] Registered new tserver with Master: 6523307100334832b7fa15fd316ff9ea (127.10.109.65:42795)
I20260620 18:34:52.806205 18851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:36931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:34:52.822995 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:52.903103 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:52.903282 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:52.903304 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:52.904350 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:52.904381 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:52.904405 19047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:52.905682 19047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:52.905856 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:52.906011 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:34:52.911085 19047 server_base.cc:1034] running on GCE node
W20260620 18:34:52.911154 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.911180 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:52.911273 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:52.911533 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:52.911767 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:52.912957 19047 hybrid_clock.cc:648] HybridClock initialized: now 1781980492912956 us; error 39 us; skew 500 ppm
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:52.915491 19047 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:34:52.915840 19047 webserver.cc:460] Webserver started at http://127.10.109.66:40641/ using document root <none> and password file <none>
I20260620 18:34:52.916006 19047 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:52.916043 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:52.916153 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:52.917073 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "1aa02c6b40e24ca18bd67a17a51627dd"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "62c36c29d17dbe4623977cc157683d8e"
server_key_iv: "f3cb068ea3ec3cb930f4f25f51a26999"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.917362 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "1aa02c6b40e24ca18bd67a17a51627dd"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "62c36c29d17dbe4623977cc157683d8e"
server_key_iv: "f3cb068ea3ec3cb930f4f25f51a26999"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.918612 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:34:52.919363 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.919516 19047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:52.919598 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "1aa02c6b40e24ca18bd67a17a51627dd"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "62c36c29d17dbe4623977cc157683d8e"
server_key_iv: "f3cb068ea3ec3cb930f4f25f51a26999"
server_key_version: "encryptionkey@0"
I20260620 18:34:52.919649 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:52.927577 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:52.929318 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:52.929420 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:52.929582 19047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:52.929867 19047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:52.929899 19047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.929919 19047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:52.929932 19047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:52.935398 19047 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:41079
I20260620 18:34:52.935473 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:41079 every 8 connection(s)
I20260620 18:34:52.935774 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:52.937908 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19047
I20260620 18:34:52.938041 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:52.938277 10677 external_mini_cluster.cc:1428] Setting key 48e94603fb57946c09bd56eb7d4217a4
Jun 20 18:34:52 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:52.945466 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:52.936530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:42761 (local address 127.10.109.126:36535)
0620 18:34:52.936576 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:52.936578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:52.936754 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:34:52.937232 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:52.937233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:52.937245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:52.937312 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:52.938239 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.938958 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.939881 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.940225 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.940297 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:52.942707 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:52.942709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:52.942709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:52.942716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:52.944097 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.944504 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.944505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.944505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.944566 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:52.944820 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:52.944821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:52.944822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:52.944931 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:52.945010 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:52.945275 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:34:52.945379 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:34:52.945765 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:52.936486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:42761)
0620 18:34:52.936634 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:52.936673 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:52.937120 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:52.937362 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:52.937365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:52.937887 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:52.938141 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:52.938146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.939043 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.939044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:52.939785 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:52.939787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:52.940318 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:52.940319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:52.940465 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:52.940939 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:52.940948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:52.942611 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:52.944199 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:52.944206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:52.944208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:52.944426 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:52.944643 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:52.944643 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:52.944644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:52.944747 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:52.945000 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:52.945007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:52.945139 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 18:34:52.945239 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:34:52.945416 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260620 18:34:52.946314 19176 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36535
I20260620 18:34:52.946417 19176 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:34:52.946609 19176 heartbeater.cc:511] Master 127.10.109.126:36535 requested a full tablet report, sending...
I20260620 18:34:52.947016 18851 ts_manager.cc:194] Registered new tserver with Master: 1aa02c6b40e24ca18bd67a17a51627dd (127.10.109.66:41079)
I20260620 18:34:52.947394 18851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:42761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:34:52.965200 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:53.046979 19181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.047164 19181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.047187 19181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.048314 19181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.048353 19181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.048408 19181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:53.049876 19181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:53.050108 19181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.050338 19181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.055024 19189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.055025 19186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.055065 19187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.055301 19181 server_base.cc:1034] running on GCE node
I20260620 18:34:53.055491 19181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.055665 19181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.056823 19181 hybrid_clock.cc:648] HybridClock initialized: now 1781980493056800 us; error 40 us; skew 500 ppm
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.059283 19181 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:34:53.059696 19181 webserver.cc:460] Webserver started at http://127.10.109.67:35591/ using document root <none> and password file <none>
I20260620 18:34:53.059926 19181 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.059970 19181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.060092 19181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:34:53.061026 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "84147d44ffe141878af539e242903384"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-3jql"
server_key: "a096eca9414903f9922097e9267c900d"
server_key_iv: "7d032e2df161266f6cec450d425ae0be"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.061378 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "84147d44ffe141878af539e242903384"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-3jql"
server_key: "a096eca9414903f9922097e9267c900d"
server_key_iv: "7d032e2df161266f6cec450d425ae0be"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.062731 19181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:34:53.063556 19196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.063779 19181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:53.063863 19181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "84147d44ffe141878af539e242903384"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-3jql"
server_key: "a096eca9414903f9922097e9267c900d"
server_key_iv: "7d032e2df161266f6cec450d425ae0be"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.063915 19181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.089867 19181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.092196 19181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.092304 19181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.092487 19181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:53.092803 19181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:53.092844 19181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.092867 19181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:53.092880 19181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.098634 19181 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:35065
I20260620 18:34:53.098691 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:35065 every 8 connection(s)
I20260620 18:34:53.099005 19181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:53.099717 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19181
I20260620 18:34:53.099828 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:53.100028 10677 external_mini_cluster.cc:1428] Setting key 8abcc6836b6329d3b80abdc30c56ba27
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:34:53.108738 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.099774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:38609 (local address 127.10.109.126:36535)
0620 18:34:53.099820 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.099822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.100124 (+ 302us) server_negotiation.cc:407] Connection header received
0620 18:34:53.100738 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.100739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.100751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.100807 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.101786 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.102282 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.103294 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.103491 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.103577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.105760 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:53.105762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:53.105762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:53.105770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:53.107280 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.107711 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.107712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.107712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.107773 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.108039 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.108040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.108040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.108154 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:53.108226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:53.108560 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:34:53.108649 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:34:53.108908 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.099749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:38609)
0620 18:34:53.099968 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.100002 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.100644 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.100872 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.100876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.101500 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.101679 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.101683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.102378 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.102380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.103194 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.103197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.103586 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.103587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.103723 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.104167 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.104175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.105659 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:53.107387 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:53.107394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:53.107396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:53.107617 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:53.107860 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:53.107861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:53.107861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:53.107969 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:53.108229 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:53.108236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:53.108397 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:34:53.108528 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 18:34:53.108677 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260620 18:34:53.109351 19310 heartbeater.cc:347] Connected to a master server at 127.10.109.126:36535
I20260620 18:34:53.109463 19310 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:53.109635 19310 heartbeater.cc:511] Master 127.10.109.126:36535 requested a full tablet report, sending...
I20260620 18:34:53.109988 18851 ts_manager.cc:194] Registered new tserver with Master: 84147d44ffe141878af539e242903384 (127.10.109.67:35065)
I20260620 18:34:53.110317 18851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:38609
I20260620 18:34:53.111999 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
Entry for principal oryx/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab.
WARNING: no policy specified for oryx/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
Entry for principal oryx/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab.
WARNING: no policy specified for oryx/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
Entry for principal oryx/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab.
I20260620 18:34:53.183166 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18913
I20260620 18:34:53.187639 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19047
I20260620 18:34:53.191972 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19181
I20260620 18:34:53.196002 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 18819
I20260620 18:34:53.200866 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36535
--webserver_interface=127.10.109.126
--webserver_port=42829
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:53.279772 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.279994 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.280028 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.281139 19319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:53.281216 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.281243 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.281272 19319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:53.281294 19319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:53.282840 19319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:53.283092 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.283316 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.288038 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.288051 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.288103 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.288445 19319 server_base.cc:1034] running on GCE node
I20260620 18:34:53.288681 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.288975 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.290146 19319 hybrid_clock.cc:648] HybridClock initialized: now 1781980493290132 us; error 43 us; skew 500 ppm
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.292575 19319 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:34:53.293004 19319 webserver.cc:460] Webserver started at http://127.10.109.126:42829/ using document root <none> and password file <none>
I20260620 18:34:53.293241 19319 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.293295 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.294628 19319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:34:53.295379 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.295583 19319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:53.295679 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "67e80c0629d8d4f70dc856c91822d77f"
server_key_iv: "54abddb5113f2c433b96f43ccd0abe4d"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.295992 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.302951 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.305054 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.305191 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.309366 19319 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36535
I20260620 18:34:53.309407 19386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36535 every 8 connection(s)
I20260620 18:34:53.309775 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:53.311278 19387 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:34:53.311928 19387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Bootstrap starting.
I20260620 18:34:53.315510 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19319
I20260620 18:34:53.315862 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:42795
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:36535
--webserver_port=40073
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:34:53.320604 19387 log.cc:824] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:53.322314 19387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:34:53.322610 19387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Bootstrap complete.
I20260620 18:34:53.324937 19387 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:53.325204 19387 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bcd83d12d844d5b1aaadab3e1a54f3, State: Initialized, Role: FOLLOWER
I20260620 18:34:53.325369 19387 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:53.325439 19387 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:53.325486 19387 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:53.325533 19387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:34:53.326480 19387 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:53.326560 19387 leader_election.cc:302] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5bcd83d12d844d5b1aaadab3e1a54f3; no voters:
I20260620 18:34:53.326785 19387 leader_election.cc:288] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:34:53.326821 19391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:34:53.326939 19391 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [term 2 LEADER]: Becoming Leader. State: Replica: e5bcd83d12d844d5b1aaadab3e1a54f3, State: Running, Role: LEADER
I20260620 18:34:53.327020 19391 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } }
I20260620 18:34:53.327049 19387 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:53.327701 19392 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5bcd83d12d844d5b1aaadab3e1a54f3. Latest consensus state: current_term: 2 leader_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } } }
I20260620 18:34:53.327778 19392 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:53.328048 19393 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bcd83d12d844d5b1aaadab3e1a54f3" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36535 } } }
I20260620 18:34:53.328104 19393 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:53.328094 19400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:53.328660 19400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:53.328799 19400 catalog_manager.cc:1259] Loaded cluster ID: 425425e58d564162a51998d199677d5b
I20260620 18:34:53.329053 19400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:53.329942 19400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:53.330127 19400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5bcd83d12d844d5b1aaadab3e1a54f3: Loaded TSK: 0
I20260620 18:34:53.330348 19400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:34:53.428812 19389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.429095 19389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.429133 19389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.430791 19389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.430908 19389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.430965 19389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:34:53.432986 19389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.65.keytab
--principal=oryx/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:42795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=40073
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:53.433221 19389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.433477 19389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.439013 19414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.439088 19389 server_base.cc:1034] running on GCE node
W20260620 18:34:53.439136 19413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.439014 19416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.439527 19389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.439798 19389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.441020 19389 hybrid_clock.cc:648] HybridClock initialized: now 1781980493440976 us; error 30 us; skew 500 ppm
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.443768 19389 init.cc:377] Logged in from keytab as oryx/127.10.109.65@KRBTEST.COM (short username oryx)
I20260620 18:34:53.444765 19389 webserver.cc:460] Webserver started at http://127.10.109.65:40073/ using document root <none> and password file <none>
I20260620 18:34:53.445050 19389 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.445170 19389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.446583 19389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:34:53.447310 19423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.447515 19389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:53.447614 19389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "6523307100334832b7fa15fd316ff9ea"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "9f71ef8d7b83478b2e4c99718fbc4d02"
server_key_iv: "56db30c6b49976e0cbed9bf96e75a74c"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.447974 19389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.456615 19389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.458588 19389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.458806 19389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.459033 19389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:53.459326 19389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:53.459379 19389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.459416 19389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:53.459443 19389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.465304 19389 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:42795
I20260620 18:34:53.465354 19536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:42795 every 8 connection(s)
I20260620 18:34:53.465673 19389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:53.471601 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19389
I20260620 18:34:53.471943 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:41079
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:36535
--webserver_port=40641
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.475438 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.466333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:33577)
0620 18:34:53.466524 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.466569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.467108 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.467536 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.467538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.468077 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.468252 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.468256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.469823 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.469824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.470581 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.470583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.471456 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.471457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.471609 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.472161 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.472170 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.475254 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
W20260620 18:34:53.475844 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.466427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:33577 (local address 127.10.109.126:36535)
0620 18:34:53.466862 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.466872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.466915 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:34:53.467300 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.467307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.467361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.467502 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.468373 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.469673 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.470674 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.471314 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.471458 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.475630 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:33577: BlockingRecv error: recv got EOF from 127.10.109.65:33577 (error 108)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":86,"threads_started":1}
W20260620 18:34:53.475880 19537 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.109.126:36535: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.481694 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.476093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:55571)
0620 18:34:53.476329 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.476341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.476395 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.476548 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.476549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.476698 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.476748 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.476750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.477719 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.477720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.478504 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.478507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.479061 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.479063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.479134 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.479570 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.479575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.481628 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":60,"threads_started":1}
W20260620 18:34:53.481824 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.476148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:55571 (local address 127.10.109.126:36535)
0620 18:34:53.476184 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.476186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.476400 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:34:53.476448 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.476450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.476462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.476533 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.476828 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.477622 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.478600 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.478973 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.479053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.481740 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:55571: BlockingRecv error: recv got EOF from 127.10.109.65:55571 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.487072 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.482072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:55485)
0620 18:34:53.482113 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.482123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.482156 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.482344 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.482345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.482483 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.482513 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.482514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.483456 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.483457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.484247 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.484250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.484715 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.484716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.484765 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.485192 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.485197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.487010 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:34:53.487185 19537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:34:53.487385 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.482118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:55485 (local address 127.10.109.126:36535)
0620 18:34:53.482150 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.482152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.482162 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:34:53.482244 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.482245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.482257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.482307 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.482597 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.483361 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.484342 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.484601 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.484666 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.487333 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:55485: BlockingRecv error: recv got EOF from 127.10.109.65:55485 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 18:34:53.554505 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.554723 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.554764 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.555964 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.556010 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.556036 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:34:53.557471 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.66.keytab
--principal=oryx/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:41079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=40641
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:53.557718 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.557950 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.563053 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.563047 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.563048 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.563268 19541 server_base.cc:1034] running on GCE node
I20260620 18:34:53.563534 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.563853 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.565016 19541 hybrid_clock.cc:648] HybridClock initialized: now 1781980493565007 us; error 38 us; skew 500 ppm
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.567728 19541 init.cc:377] Logged in from keytab as oryx/127.10.109.66@KRBTEST.COM (short username oryx)
I20260620 18:34:53.568171 19541 webserver.cc:460] Webserver started at http://127.10.109.66:40641/ using document root <none> and password file <none>
I20260620 18:34:53.568415 19541 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.568497 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.569880 19541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:53.570415 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.570585 19541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:34:53.570683 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "1aa02c6b40e24ca18bd67a17a51627dd"
format_stamp: "Formatted at 2026-06-20 18:34:52 on dist-test-slave-3jql"
server_key: "62c36c29d17dbe4623977cc157683d8e"
server_key_iv: "f3cb068ea3ec3cb930f4f25f51a26999"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.571086 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.593720 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.595974 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.596136 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.596352 19541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:53.596645 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:53.596696 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.596731 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:53.596757 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.602488 19541 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:41079
I20260620 18:34:53.602563 19670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:41079 every 8 connection(s)
I20260620 18:34:53.602936 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:53.607213 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19541
I20260620 18:34:53.607542 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:35065
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:36535
--webserver_port=35591
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:34:53.609961 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.603601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:43091 (local address 127.10.109.126:36535)
0620 18:34:53.603646 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.603648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.603836 (+ 188us) server_negotiation.cc:407] Connection header received
0620 18:34:53.604296 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.604297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.604312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.604360 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.605214 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.605703 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.606655 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.606908 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.606961 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.609904 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:43091: BlockingRecv error: recv got EOF from 127.10.109.66:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:34:53.610018 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.603598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:43091)
0620 18:34:53.603732 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.603764 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.604209 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.604424 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.604427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.604966 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.605140 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.605144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.605779 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.605779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.606583 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.606585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.606978 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.606979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.607099 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.607552 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.607559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.609832 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260620 18:34:53.610214 19671 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.109.126:36535: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.614602 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.610442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:38199)
0620 18:34:53.610489 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.610500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.610542 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.610673 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.610673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.610839 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.610879 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.610880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.611499 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.611500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.611988 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.611990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.612366 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.612367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.612431 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.612856 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.612860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.614525 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:34:53.614657 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.610493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:38199 (local address 127.10.109.126:36535)
0620 18:34:53.610535 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.610537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.610546 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:53.610606 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.610606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.610615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.610669 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.610955 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.611407 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.612068 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.612256 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.612354 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.614593 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:38199: BlockingRecv error: recv got EOF from 127.10.109.66:38199 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.618950 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.614932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:48719)
0620 18:34:53.614985 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.614999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.615051 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.615168 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.615168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.615298 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.615320 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.615322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.615915 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.615917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.616390 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.616392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.616725 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.616726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.616777 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.617185 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.617189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.618889 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 18:34:53.619040 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.614999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:48719 (local address 127.10.109.126:36535)
0620 18:34:53.615057 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.615059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.615084 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:34:53.615103 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.615104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.615112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.615163 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.615399 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.615839 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.616469 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.616654 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.616727 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.618962 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:48719: BlockingRecv error: recv got EOF from 127.10.109.66:48719 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 18:34:53.619051 19671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:34:53.694692 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:53.694960 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:53.694983 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:53.696135 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:53.696174 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:53.696200 19674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:34:53.697600 19674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/krb5kdc/oryx_127.10.109.67.keytab
--principal=oryx/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:35065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=35591
--tserver_master_addrs=127.10.109.126:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:34:53.697826 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:53.698045 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:53.703007 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.703071 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:53.703001 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:53.703145 19674 server_base.cc:1034] running on GCE node
I20260620 18:34:53.703447 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:53.703683 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:53.704872 19674 hybrid_clock.cc:648] HybridClock initialized: now 1781980493704865 us; error 34 us; skew 500 ppm
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980493, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:53.707322 19674 init.cc:377] Logged in from keytab as oryx/127.10.109.67@KRBTEST.COM (short username oryx)
I20260620 18:34:53.707677 19674 webserver.cc:460] Webserver started at http://127.10.109.67:35591/ using document root <none> and password file <none>
I20260620 18:34:53.707911 19674 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:53.707995 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:53.709290 19674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:34:53.709947 19689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.710148 19674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:34:53.710247 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "84147d44ffe141878af539e242903384"
format_stamp: "Formatted at 2026-06-20 18:34:53 on dist-test-slave-3jql"
server_key: "a096eca9414903f9922097e9267c900d"
server_key_iv: "7d032e2df161266f6cec450d425ae0be"
server_key_version: "encryptionkey@0"
I20260620 18:34:53.710592 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:53.733143 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:53.735106 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:53.735273 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:53.735517 19674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:53.735834 19674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:53.735890 19674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.735932 19674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:53.735960 19674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:53.741647 19674 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:35065
I20260620 18:34:53.741771 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:35065 every 8 connection(s)
I20260620 18:34:53.742059 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:53.743183 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19674
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:34:53.750164 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.742854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:57437 (local address 127.10.109.126:36535)
0620 18:34:53.742904 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.742907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.743122 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:34:53.743645 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.743647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.743658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.743716 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.744772 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.745271 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.746321 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.746593 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.746664 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.750119 (+ 3455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:57437: BlockingRecv error: recv got EOF from 127.10.109.67:57437 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:34:53.750247 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.742816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:57437)
0620 18:34:53.742977 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.743023 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.743538 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.743853 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.743857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.744482 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.744657 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.744661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.745362 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.745363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.746228 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.746231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.746950 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.746951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.747056 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.747504 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.747512 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.750062 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260620 18:34:53.750480 19803 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:36535 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.109.126:36535: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
I20260620 18:34:53.751323 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.743729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.10.109.126:36535)
0620 18:34:53.743934 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.743937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.743945 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:53.743964 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.743965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.743974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.744048 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.744777 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.745278 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.746047 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.746366 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.746416 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.747634 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:53.747638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:53.747638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:53.747648 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:53.749527 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.749886 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.749889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.749889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.750003 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:53.750462 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:53.750464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:53.750464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:53.750794 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:53.750971 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:53.751121 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:34:53.751236 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":45,"threads_started":1}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.754974 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.750672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:53453)
0620 18:34:53.750721 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.750730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.750790 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.750914 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.750914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.751076 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.751126 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.751128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.751880 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.751881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.752478 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.752480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.752828 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.752828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.752893 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.753273 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.753277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.754899 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 18:34:53.755016 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.750718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:53453 (local address 127.10.109.126:36535)
0620 18:34:53.750773 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.750775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.750783 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:53.750850 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.750851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.750860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.750913 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.751199 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.751786 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.752567 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.752757 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.752827 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.754962 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:53453: BlockingRecv error: recv got EOF from 127.10.109.67:53453 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:53 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:53.758611 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:53.755254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:38597)
0620 18:34:53.755294 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:53.755303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:53.755334 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:53.755431 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:53.755432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:53.755528 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:53.755550 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.755550 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.756073 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.756074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.756533 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:53.756534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.756815 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.756816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:53.756849 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.757143 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:53.757146 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:53.758573 (+ 1427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:34:53.758724 19803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:34:53.758822 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:53.755296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:38597 (local address 127.10.109.126:36535)
0620 18:34:53.755338 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:53.755340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:53.755360 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:34:53.755384 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:53.755385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:53.755392 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:53.755429 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:53.755591 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.756012 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.756599 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:53.756758 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:53.756804 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:53.758722 (+ 1918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:38597: BlockingRecv error: recv got EOF from 127.10.109.67:38597 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:54.492362 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:54.487575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:47063)
0620 18:34:54.487718 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:54.487828 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:54.487890 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:54.488037 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:54.488038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:54.488198 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:54.488267 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.488269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.488927 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.488928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.489462 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.489463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.489862 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.489863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.489922 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.490459 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:54.490463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:54.492291 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
W20260620 18:34:54.492504 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:54.487770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:47063 (local address 127.10.109.126:36535)
0620 18:34:54.487903 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:54.487905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:54.487913 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:34:54.487933 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:54.487933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:54.487942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:54.488004 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:54.488338 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.488840 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.489543 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.489748 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.489807 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.492390 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:47063: BlockingRecv error: recv got EOF from 127.10.109.65:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:54.624006 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:54.619441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:55559)
0620 18:34:54.619571 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:54.619589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:54.619655 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:54.619788 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:54.619789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:54.619948 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:54.619995 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.619996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.620671 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.620673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.621224 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.621226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.621584 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.621585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.621629 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.622048 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:54.622053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:54.623932 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260620 18:34:54.624064 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:54.619496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:55559 (local address 127.10.109.126:36535)
0620 18:34:54.619571 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:54.619573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:54.619653 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:34:54.619713 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:54.619713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:54.619724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:54.619767 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:54.620061 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.620556 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.621308 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.621511 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.621568 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.624019 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:55559: BlockingRecv error: recv got EOF from 127.10.109.66:55559 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:54 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:54.763638 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:54.759134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:39985)
0620 18:34:54.759278 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:54.759297 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:54.759353 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:54.759463 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:54.759463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:54.759618 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:54.759667 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.759668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.760331 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.760348 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.760900 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:54.760902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.761238 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.761239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:54.761277 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.761705 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:54.761711 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:54.763572 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
W20260620 18:34:54.763716 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:54.759196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39985 (local address 127.10.109.126:36535)
0620 18:34:54.759244 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:54.759246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:54.759340 (+ 94us) server_negotiation.cc:407] Connection header received
0620 18:34:54.759405 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:54.759406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:54.759418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:54.759457 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:54.759742 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.760240 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.760984 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:54.761171 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:54.761228 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:54.763668 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:39985: BlockingRecv error: recv got EOF from 127.10.109.67:39985 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:55.497684 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:55.492862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:37515)
0620 18:34:55.493018 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:55.493029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:55.493077 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:55.493296 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:55.493297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:55.493453 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:55.493507 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.493509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.494273 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.494274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.494896 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.494898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.495235 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.495236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.495275 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.495927 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:55.495931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:55.497626 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260620 18:34:55.497865 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:55.492927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:37515 (local address 127.10.109.126:36535)
0620 18:34:55.493073 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:55.493075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:55.493087 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:34:55.493113 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:55.493113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:55.493124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:55.493294 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:55.493629 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.494186 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.494977 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.495163 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.495221 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.497739 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:37515: BlockingRecv error: recv got EOF from 127.10.109.65:37515 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:55.629518 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:55.624519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:54631)
0620 18:34:55.624727 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:55.624739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:55.624834 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:55.625010 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:55.625011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:55.625190 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:55.625234 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.625236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.625915 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.625916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.626450 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.626452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.626877 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.626878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.626944 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.627499 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:55.627511 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:55.629455 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
W20260620 18:34:55.629690 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:55.624625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:54631 (local address 127.10.109.126:36535)
0620 18:34:55.624712 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:55.624715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:55.624813 (+ 98us) server_negotiation.cc:407] Connection header received
0620 18:34:55.624922 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:55.624923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:55.624935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:55.625004 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:55.625330 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.625832 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.626539 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.626785 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.626876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.629552 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:54631: BlockingRecv error: recv got EOF from 127.10.109.66:54631 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:55 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:55.769265 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:55.764284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:53789)
0620 18:34:55.764448 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:55.764465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:55.764550 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:55.764682 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:55.764683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:55.764862 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:55.764908 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.764910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.765625 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.765626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.766137 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:55.766139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.766526 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.766527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:55.766592 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.767130 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:55.767135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:55.769131 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
W20260620 18:34:55.769265 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:55.764454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:53789 (local address 127.10.109.126:36535)
0620 18:34:55.764529 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:55.764531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:55.764543 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:34:55.764612 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:55.764613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:55.764628 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:55.764675 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:55.764993 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.765518 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.766224 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:55.766417 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:55.766528 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:55.769213 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:53789: BlockingRecv error: recv got EOF from 127.10.109.67:53789 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:56.502897 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:56.498160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:47105)
0620 18:34:56.498317 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:56.498327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:56.498374 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:56.498535 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:56.498536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:56.498691 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:56.498792 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.498795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.499602 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.499603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.500138 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.500140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.500490 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.500491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.500530 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.501040 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:56.501045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:56.502838 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
W20260620 18:34:56.503048 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:56.498231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:47105 (local address 127.10.109.126:36535)
0620 18:34:56.498369 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:56.498371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:56.498382 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:56.498419 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:56.498419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:56.498429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:56.498508 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:56.498897 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.499474 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.500221 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.500426 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.500482 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.502957 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:47105: BlockingRecv error: recv got EOF from 127.10.109.65:47105 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:56.635108 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:56.630046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:43903)
0620 18:34:56.630346 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:56.630357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:56.630425 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:56.630688 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:56.630690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:56.630923 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:56.630970 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.630972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.631640 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.631641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.632160 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.632162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.632513 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.632514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.632563 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.633019 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:56.633024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:56.635011 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":224,"threads_started":1}
W20260620 18:34:56.635146 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:56.630240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:43903 (local address 127.10.109.126:36535)
0620 18:34:56.630361 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:56.630363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:56.630427 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:34:56.630543 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:56.630544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:56.630556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:56.630619 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:56.631054 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.631547 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.632239 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.632432 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.632488 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.635087 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:43903: BlockingRecv error: recv got EOF from 127.10.109.66:43903 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:56 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:56.774598 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:56.769860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:42861)
0620 18:34:56.769975 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:56.769987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:56.770045 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:56.770160 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:56.770161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:56.770337 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:56.770381 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.770383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.771083 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.771084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.771619 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:56.771620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.772123 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.772124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:56.772175 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.772725 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:56.772730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:56.774537 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260620 18:34:56.774858 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:56.769938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:42861 (local address 127.10.109.126:36535)
0620 18:34:56.770040 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:56.770042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:56.770053 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:56.770083 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:56.770083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:56.770102 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:56.770155 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:56.770472 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.771005 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.771759 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:56.772010 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:56.772077 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:56.774739 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:42861: BlockingRecv error: recv got EOF from 127.10.109.67:42861 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:57.508054 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:57.503369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:49201)
0620 18:34:57.503519 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:57.503530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:57.503579 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:57.503733 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:57.503733 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:57.503905 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:57.503953 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.503955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.504639 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.504641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.505197 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.505199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.505566 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.505567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.505618 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.506170 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:57.506175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:57.507995 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260620 18:34:57.508168 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:57.503456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:49201 (local address 127.10.109.126:36535)
0620 18:34:57.503576 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:57.503578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:57.503587 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:34:57.503637 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:57.503638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:57.503648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:57.503723 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:57.504027 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.504547 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.505280 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.505471 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.505541 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.508086 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:49201: BlockingRecv error: recv got EOF from 127.10.109.65:49201 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:57.644519 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:57.635792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:45427)
0620 18:34:57.636010 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:57.636024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:57.636106 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:57.636331 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:57.636333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:57.636506 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:57.636560 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.636561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.637254 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.637255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.637807 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.637809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.638156 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.638157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.638195 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.642605 (+ 4410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:57.642610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:57.644458 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":147,"threads_started":1}
W20260620 18:34:57.644739 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:57.635904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:45427 (local address 127.10.109.126:36535)
0620 18:34:57.635957 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:57.635959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:57.636113 (+ 154us) server_negotiation.cc:407] Connection header received
0620 18:34:57.636177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:57.636178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:57.636190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:57.636246 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:57.636642 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.637179 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.637897 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.638086 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.638149 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.644577 (+ 6428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:45427: BlockingRecv error: recv got EOF from 127.10.109.66:45427 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:57 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:57.779913 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:57.775071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:40321)
0620 18:34:57.775223 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:57.775234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:57.775301 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:57.775441 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:57.775442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:57.775612 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:57.775659 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.775661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.776315 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.776316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.776829 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:57.776831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.777329 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.777330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:57.777379 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.777935 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:57.777940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:57.779850 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
W20260620 18:34:57.780076 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:57.775150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:40321 (local address 127.10.109.126:36535)
0620 18:34:57.775233 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:57.775235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:57.775302 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:34:57.775357 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:57.775358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:57.775371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:57.775425 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:57.775729 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.776227 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.776968 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:57.777215 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:57.777315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:57.779996 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:40321: BlockingRecv error: recv got EOF from 127.10.109.67:40321 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:58.513406 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:58.508558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:44595)
0620 18:34:58.508702 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:58.508720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:58.508774 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:58.508937 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:58.508939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:58.509094 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:58.509143 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.509145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.509906 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.509907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.510571 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.510573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.511015 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.511016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.511076 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.511501 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:58.511506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:58.513302 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260620 18:34:58.513450 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:58.508632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:44595 (local address 127.10.109.126:36535)
0620 18:34:58.508760 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:58.508762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:58.508773 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:34:58.508820 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:58.508821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:58.508831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:58.508899 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:58.509218 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.509830 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.510654 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.510941 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.511017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.513383 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:44595: BlockingRecv error: recv got EOF from 127.10.109.65:44595 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:58.649448 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:58.645020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:33065)
0620 18:34:58.645165 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:58.645175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:58.645221 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:58.645350 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:58.645351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:58.645504 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:58.645550 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.645552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.646205 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.646207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.646820 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.646822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.647156 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.647157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.647198 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.647706 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:58.647710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:58.649391 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260620 18:34:58.649616 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:58.645141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:33065 (local address 127.10.109.126:36535)
0620 18:34:58.645194 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:58.645198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:58.645210 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:34:58.645278 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:58.645279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:58.645289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:58.645331 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:58.645615 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.646095 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.646910 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.647089 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.647146 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.649534 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:33065: BlockingRecv error: recv got EOF from 127.10.109.66:33065 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:58 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:58.785276 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:58.780393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:54837)
0620 18:34:58.780706 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:58.780717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:58.780800 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:58.780965 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:58.780965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:58.781180 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:58.781225 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.781227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.781874 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.781876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.782491 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:58.782493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.782887 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.782888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:58.782948 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.783384 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:58.783389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:58.785179 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":84,"threads_started":1}
W20260620 18:34:58.785315 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:58.780479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:54837 (local address 127.10.109.126:36535)
0620 18:34:58.780658 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:58.780660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:58.780783 (+ 123us) server_negotiation.cc:407] Connection header received
0620 18:34:58.780856 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:58.780857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:58.780871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:58.780945 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:58.781304 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.781793 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.782550 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:58.782797 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:58.782885 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:58.785254 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:54837: BlockingRecv error: recv got EOF from 127.10.109.67:54837 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:59.518965 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:59.513921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:58317)
0620 18:34:59.514084 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:59.514099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:59.514165 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:59.514368 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:59.514369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:59.514548 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:59.514602 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.514603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.515287 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.515288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.515823 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.515824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.516180 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.516180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.516220 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.516685 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:59.516689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:59.518908 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260620 18:34:59.519104 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:59.513986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:58317 (local address 127.10.109.126:36535)
0620 18:34:59.514098 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:59.514099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:59.514148 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:34:59.514259 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:59.514260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:59.514275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:59.514328 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:59.514690 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.515221 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.515916 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.516108 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.516170 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.519007 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:58317: BlockingRecv error: recv got EOF from 127.10.109.65:58317 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:59.655977 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:59.649957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:34175)
0620 18:34:59.650129 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:59.650142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:59.650209 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:59.650393 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:59.650394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:59.650662 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:59.650720 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.650723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.651399 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.651400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.652303 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.652306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.652756 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.652759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.652856 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.653459 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:59.653465 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:59.655913 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
W20260620 18:34:59.656145 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:59.650083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:34175 (local address 127.10.109.126:36535)
0620 18:34:59.650159 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:59.650161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:59.650191 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:34:59.650295 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:59.650296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:59.650308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:59.650375 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:59.650823 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.651320 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.652403 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.652606 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.652668 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.656047 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:34175: BlockingRecv error: recv got EOF from 127.10.109.66:34175 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:34:59 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:34:59.790671 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:59.785826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:39437)
0620 18:34:59.785970 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:59.785981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:59.786047 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:59.786284 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:59.786287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:59.786674 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:59.786727 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.786730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.787432 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.787433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.787982 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:59.787984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.788344 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.788345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:59.788386 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.788919 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:59.788923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:59.790611 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260620 18:34:59.790946 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:34:59.785915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39437 (local address 127.10.109.126:36535)
0620 18:34:59.785987 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:59.785989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:59.786031 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:34:59.786098 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:59.786099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:59.786110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:59.786163 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:59.786877 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.787360 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.788083 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:59.788268 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:59.788334 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:59.790865 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:39437: BlockingRecv error: recv got EOF from 127.10.109.67:39437 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:00.524181 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:00.519511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:50087)
0620 18:35:00.519803 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:00.519813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:00.519867 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:00.520001 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:00.520002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:00.520162 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:00.520209 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.520211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.520910 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.520911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.521431 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.521433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.521799 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.521800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.521840 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.522352 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:00.522357 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:00.524123 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":200,"threads_started":1}
W20260620 18:35:00.524371 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:00.519689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:50087 (local address 127.10.109.126:36535)
0620 18:35:00.519820 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:00.519822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:00.519854 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:35:00.519922 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:00.519922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:00.519933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:00.519984 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:00.520303 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.520833 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.521514 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.521730 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.521788 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.524264 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:50087: BlockingRecv error: recv got EOF from 127.10.109.65:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:00.660904 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:00.656493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:48281)
0620 18:35:00.656656 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:00.656667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:00.656718 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:00.656835 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:00.656836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:00.656990 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:00.657036 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.657038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.657706 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.657707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.658226 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.658227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.658557 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.658558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.658598 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.659082 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:00.659087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:00.660837 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260620 18:35:00.660979 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:00.656646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:48281 (local address 127.10.109.126:36535)
0620 18:35:00.656710 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:00.656712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:00.656722 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:35:00.656756 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:00.656757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:00.656767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:00.656817 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:00.657102 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.657606 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.658310 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.658489 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.658546 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.660932 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:48281: BlockingRecv error: recv got EOF from 127.10.109.66:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:00 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:00.796046 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:00.791161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:58495)
0620 18:35:00.791477 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:00.791490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:00.791562 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:00.791730 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:00.791731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:00.791927 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:00.791974 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.791976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.792666 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.792667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.793230 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:00.793231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.793600 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.793601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:00.793650 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.794265 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:00.794270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:00.795987 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":57,"threads_started":1}
W20260620 18:35:00.796356 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:00.791354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:58495 (local address 127.10.109.126:36535)
0620 18:35:00.791427 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:00.791429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:00.791585 (+ 156us) server_negotiation.cc:407] Connection header received
0620 18:35:00.791614 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:00.791615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:00.791628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:00.791699 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:00.792067 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.792583 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.793326 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:00.793509 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:00.793570 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:00.796268 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:58495: BlockingRecv error: recv got EOF from 127.10.109.67:58495 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:01.529615 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:01.524676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:44889)
0620 18:35:01.524864 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:01.524874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:01.524924 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:01.525045 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:01.525046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:01.525211 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:01.525259 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.525260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.525961 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.525961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.526509 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.526511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.526935 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.526935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.526978 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.527568 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:01.527575 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:01.529558 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
W20260620 18:35:01.529762 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:01.524812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:44889 (local address 127.10.109.126:36535)
0620 18:35:01.524936 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:01.524938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:01.524945 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:35:01.524966 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:01.524967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:01.524977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:01.525034 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:01.525347 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.525884 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.526572 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.526865 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.526927 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.529670 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:44889: BlockingRecv error: recv got EOF from 127.10.109.65:44889 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:01.666193 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:01.661513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:34305)
0620 18:35:01.661665 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:01.661679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:01.661745 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:01.661905 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:01.661906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:01.662082 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:01.662124 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.662126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.662972 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.662973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.663490 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.663491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.663904 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.663905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.663959 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.664444 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:01.664453 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:01.666096 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260620 18:35:01.666229 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:01.661619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:34305 (local address 127.10.109.126:36535)
0620 18:35:01.661688 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:01.661690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:01.661726 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:35:01.661815 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:01.661816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:01.661827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:01.661900 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:01.662202 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.662896 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.663585 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.663782 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.663905 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.666169 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:34305: BlockingRecv error: recv got EOF from 127.10.109.66:34305 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:01 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:01.802341 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:01.796557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:51709)
0620 18:35:01.796772 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:01.796798 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:01.796898 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:01.797183 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:01.797184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:01.797367 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:01.797417 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.797419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.798080 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.798081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.798634 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:01.798636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.799173 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.799174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:01.799239 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.799840 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:01.799844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:01.802269 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":57,"threads_started":1}
W20260620 18:35:01.802479 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:01.796798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:51709 (local address 127.10.109.126:36535)
0620 18:35:01.796866 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:01.796868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:01.796887 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:01.797044 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:01.797045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:01.797060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:01.797114 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:01.797497 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.798001 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.798801 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:01.799073 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:01.799141 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:01.802370 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:51709: BlockingRecv error: recv got EOF from 127.10.109.67:51709 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:02.534900 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:02.530073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:54835)
0620 18:35:02.530329 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:02.530341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:02.530388 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:02.530536 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:02.530536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:02.530700 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:02.530823 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.530830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.531477 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.531478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.531997 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.531999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.532402 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.532403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.532442 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.532913 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:02.532917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:02.534808 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":192,"threads_started":1}
W20260620 18:35:02.534986 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:02.530254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:54835 (local address 127.10.109.126:36535)
0620 18:35:02.530364 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:02.530365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:02.530389 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:02.530456 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:02.530457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:02.530468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:02.530519 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:02.530898 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.531405 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.532106 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.532329 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.532395 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.534908 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:54835: BlockingRecv error: recv got EOF from 127.10.109.65:54835 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:02.675362 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:02.666913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:40105)
0620 18:35:02.667144 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:02.667159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:02.667224 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:02.667345 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:02.667345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:02.667509 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:02.667556 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.667557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.668198 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.668199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.668854 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.668857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.669373 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.669374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.669421 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.672985 (+ 3564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:02.672991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:02.675283 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":176,"threads_started":1}
W20260620 18:35:02.675460 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:02.667030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:40105 (local address 127.10.109.126:36535)
0620 18:35:02.667092 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:02.667095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:02.667227 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:35:02.667275 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:02.667276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:02.667286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:02.667339 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:02.667614 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.668126 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.668955 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.669279 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.669362 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.675389 (+ 6027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:40105: BlockingRecv error: recv got EOF from 127.10.109.66:40105 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:02 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:02.807672 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:02.802894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:51721)
0620 18:35:02.803144 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:02.803154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:02.803227 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:02.803367 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:02.803368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:02.803541 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:02.803587 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.803589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.804292 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.804294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.804810 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:02.804811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.805185 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.805186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:02.805243 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.805713 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:02.805718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:02.807612 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260620 18:35:02.807767 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:02.803016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:51721 (local address 127.10.109.126:36535)
0620 18:35:02.803081 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:02.803084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:02.803209 (+ 125us) server_negotiation.cc:407] Connection header received
0620 18:35:02.803276 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:02.803277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:02.803294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:02.803348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:02.803673 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.804194 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.804904 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:02.805100 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:02.805185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:02.807698 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:51721: BlockingRecv error: recv got EOF from 127.10.109.67:51721 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:03.540494 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.535448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:50951)
0620 18:35:03.535620 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:03.535630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:03.535687 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:03.535816 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:03.535816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:03.535977 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:03.536026 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.536027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.537130 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.537131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.537646 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.537648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.538038 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.538039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.538084 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.538572 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:03.538577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:03.540423 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
W20260620 18:35:03.540567 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:03.535541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:50951 (local address 127.10.109.126:36535)
0620 18:35:03.535685 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.535687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.535696 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:03.535724 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.535725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.535736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.535799 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.536101 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.537048 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.537764 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.537965 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.538028 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.540509 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:50951: BlockingRecv error: recv got EOF from 127.10.109.65:50951 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:03.680406 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.675831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:38369)
0620 18:35:03.675964 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:03.675975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:03.676038 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:03.676173 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:03.676173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:03.676343 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:03.676390 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.676392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.677025 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.677026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.677798 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.677801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.678108 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.678109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.678146 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.678644 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:03.678649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:03.680347 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260620 18:35:03.680624 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:03.675912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:38369 (local address 127.10.109.126:36535)
0620 18:35:03.675976 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.675978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.676035 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:35:03.676083 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.676083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.676095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.676149 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.676451 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.676952 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.677858 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.678045 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.678100 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.680559 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:38369: BlockingRecv error: recv got EOF from 127.10.109.66:38369 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:03 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:03.813324 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.808208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:37263)
0620 18:35:03.808374 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:03.808384 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:03.808430 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:03.808710 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:03.808711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:03.808950 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:03.808999 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.809001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.809802 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.809803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.810335 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:03.810336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.810714 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.810715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:03.810812 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.811253 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:03.811258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:03.813211 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260620 18:35:03.813354 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:03.808465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:37263 (local address 127.10.109.126:36535)
0620 18:35:03.808510 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.808517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.808524 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:35:03.808548 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.808549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.808561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.808616 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.809088 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.809716 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.810432 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.810630 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.810714 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.813294 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:37263: BlockingRecv error: recv got EOF from 127.10.109.67:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:04.546608 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:04.541048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:59847)
0620 18:35:04.541262 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:04.541272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:04.541333 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:04.541471 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:04.541472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:04.541644 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:04.541687 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.541689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.542423 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.542424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.543047 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.543050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.543555 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.543557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.543607 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.544129 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:04.544134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:04.546374 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":133,"threads_started":1}
W20260620 18:35:04.546720 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:04.541167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:59847 (local address 127.10.109.126:36535)
0620 18:35:04.541273 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:04.541275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:04.541339 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:35:04.541376 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:04.541377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:04.541388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:04.541455 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:04.541796 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.542342 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.543154 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.543453 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.543522 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.546637 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:59847: BlockingRecv error: recv got EOF from 127.10.109.65:59847 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:04.685575 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:04.680842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:51923)
0620 18:35:04.681018 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:04.681029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:04.681084 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:04.681228 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:04.681228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:04.681407 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:04.681451 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.681453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.682129 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.682130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.682687 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.682689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.683136 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.683137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.683184 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.683781 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:04.683785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:04.685505 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
W20260620 18:35:04.685752 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:04.680916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:51923 (local address 127.10.109.126:36535)
0620 18:35:04.680960 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:04.680961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:04.681087 (+ 126us) server_negotiation.cc:407] Connection header received
0620 18:35:04.681147 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:04.681147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:04.681163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:04.681205 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:04.681540 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.682046 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.682840 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.683036 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.683098 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.685663 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:51923: BlockingRecv error: recv got EOF from 127.10.109.66:51923 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:04 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:04.819339 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:04.813975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:45937)
0620 18:35:04.814280 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:04.814295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:04.814381 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:04.814578 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:04.814579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:04.814847 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:04.814892 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.814895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.815557 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.815558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.816101 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:04.816103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.816499 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.816500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:04.816558 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.817159 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:04.817164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:04.819231 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":84,"threads_started":1}
W20260620 18:35:04.819368 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:04.814075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:45937 (local address 127.10.109.126:36535)
0620 18:35:04.814243 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:04.814246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:04.814361 (+ 115us) server_negotiation.cc:407] Connection header received
0620 18:35:04.814444 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:04.814445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:04.814500 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:04.814550 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:04.814979 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.815477 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.816206 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:04.816396 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:04.816484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:04.819304 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:45937: BlockingRecv error: recv got EOF from 127.10.109.67:45937 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:05.552153 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.547196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:54045)
0620 18:35:05.547364 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:05.547375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:05.547428 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:05.547609 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:05.547610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:05.547781 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:05.547826 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.547827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.548574 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.548576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.549120 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.549121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.549491 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.549492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.549569 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.550038 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:05.550044 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:05.552092 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":127,"threads_started":1}
W20260620 18:35:05.552299 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:05.547332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:54045 (local address 127.10.109.126:36535)
0620 18:35:05.547471 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.547474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.547484 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:35:05.547517 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.547518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.547531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.547605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.547899 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.548432 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.549230 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.549422 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.549479 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.552197 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:54045: BlockingRecv error: recv got EOF from 127.10.109.65:54045 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:05.690686 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.686190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:43987)
0620 18:35:05.686379 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:05.686391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:05.686453 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:05.686636 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:05.686636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:05.686872 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:05.686915 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.686918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.687532 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.687533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.688081 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.688082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.688453 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.688454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.688492 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.688985 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:05.688989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:05.690609 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
W20260620 18:35:05.690814 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:05.686314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:43987 (local address 127.10.109.126:36535)
0620 18:35:05.686365 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.686367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.686453 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:35:05.686499 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.686500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.686511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.686604 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.686985 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.687467 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.688171 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.688376 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.688447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.690698 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:43987: BlockingRecv error: recv got EOF from 127.10.109.66:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:05 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:05.824612 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.819844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:55553)
0620 18:35:05.819991 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:05.820001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:05.820058 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:05.820193 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:05.820193 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:05.820367 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:05.820408 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.820409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.821086 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.821087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.821654 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.821656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.822048 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.822049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.822266 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.822793 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:05.822798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:05.824552 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260620 18:35:05.824725 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:05.819953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:55553 (local address 127.10.109.126:36535)
0620 18:35:05.820033 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.820035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.820047 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:35:05.820108 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.820109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.820121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.820184 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.820495 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.821002 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.821752 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.821941 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.822028 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.824634 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:55553: BlockingRecv error: recv got EOF from 127.10.109.67:55553 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:06.557582 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.552638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:39231)
0620 18:35:06.552838 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:06.552849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:06.552903 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:06.553061 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:06.553062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:06.553242 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:06.553288 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.553290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.554003 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.554005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.554539 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.554540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.555054 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.555055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.555099 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.555612 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:06.555616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:06.557521 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":113,"threads_started":1}
W20260620 18:35:06.557670 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:06.552746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:39231 (local address 127.10.109.126:36535)
0620 18:35:06.552893 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.552895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.552903 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:35:06.552950 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.552951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.552961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.553020 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.553375 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.553899 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.554640 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.554982 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.555039 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.557617 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:39231: BlockingRecv error: recv got EOF from 127.10.109.65:39231 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:06.696009 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.691160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:45315)
0620 18:35:06.691331 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:06.691344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:06.691417 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:06.691562 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:06.691563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:06.691720 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:06.691765 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.691766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.692462 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.692463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.693207 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.693210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.693589 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.693590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.693646 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.694074 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:06.694078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:06.695948 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
W20260620 18:35:06.696138 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:06.691307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:45315 (local address 127.10.109.126:36535)
0620 18:35:06.691371 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.691373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.691400 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:35:06.691477 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.691478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.691488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.691537 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.691831 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.692378 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.693292 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.693506 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.693589 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.696041 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:45315: BlockingRecv error: recv got EOF from 127.10.109.66:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:06 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:06.830369 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.825094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:38425)
0620 18:35:06.825279 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:06.825290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:06.825351 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:06.825522 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:06.825523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:06.825708 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:06.825753 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.825755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.826495 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.826496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.827289 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.827292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.827640 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.827641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.827685 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.828319 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:06.828324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:06.830297 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
W20260620 18:35:06.830511 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:06.825143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:38425 (local address 127.10.109.126:36535)
0620 18:35:06.825211 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.825213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.825375 (+ 162us) server_negotiation.cc:407] Connection header received
0620 18:35:06.825402 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.825403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.825416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.825475 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.825855 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.826404 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.827367 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.827548 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.827628 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.830400 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:38425: BlockingRecv error: recv got EOF from 127.10.109.67:38425 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:07.562628 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:07.558137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:35199)
0620 18:35:07.558288 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:07.558300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:07.558356 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:07.558512 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:07.558513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:07.558688 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:07.558790 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.558793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.559429 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.559429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.559950 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.559951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.560270 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.560271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.560311 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.560778 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:07.560782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:07.562565 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
W20260620 18:35:07.562906 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:07.558262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:35199 (local address 127.10.109.126:36535)
0620 18:35:07.558389 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:07.558390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:07.558398 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:35:07.558420 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:07.558420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:07.558429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:07.558493 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:07.558869 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.559363 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.560019 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.560204 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.560255 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.562661 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:35199: BlockingRecv error: recv got EOF from 127.10.109.65:35199 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:07.707110 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:07.696459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:50981)
0620 18:35:07.696607 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:07.696618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:07.696717 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:07.696866 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:07.696867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:07.697026 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:07.697072 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.697074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.697732 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.697733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.698263 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.698268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.698625 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.698626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.698666 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.705060 (+ 6394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:07.705065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:07.707042 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260620 18:35:07.707265 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:07.696580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:50981 (local address 127.10.109.126:36535)
0620 18:35:07.696625 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:07.696627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:07.696669 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:35:07.696785 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:07.696785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:07.696795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:07.696843 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:07.697131 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.697615 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.698364 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.698554 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.698617 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.707136 (+ 8519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:50981: BlockingRecv error: recv got EOF from 127.10.109.66:50981 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:07 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.67@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:07.835271 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:07.830860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.67:48001)
0620 18:35:07.831018 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:07.831028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:07.831083 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:07.831233 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:07.831234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:07.831397 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:07.831443 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.831444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.832112 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.832113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.832640 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:07.832642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.832971 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.832972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:07.833013 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.833430 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:07.833435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:07.835196 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260620 18:35:07.835400 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:07.830926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:48001 (local address 127.10.109.126:36535)
0620 18:35:07.830999 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:07.831001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:07.831082 (+ 81us) server_negotiation.cc:407] Connection header received
0620 18:35:07.831141 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:07.831142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:07.831155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:07.831198 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:07.831508 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.832032 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.832729 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:07.832905 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:07.832958 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:07.835299 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.67:48001: BlockingRecv error: recv got EOF from 127.10.109.67:48001 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 18:35:08 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:08 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.65@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:08.567792 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.563307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.65:50365)
0620 18:35:08.563470 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:08.563480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:08.563564 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:08.563706 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:08.563707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:08.563883 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:08.563933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.563935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.564589 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.564590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.565115 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.565117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.565453 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.565453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.565496 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.565908 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:08.565913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:08.567735 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
W20260620 18:35:08.567904 19865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:08.563424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:50365 (local address 127.10.109.126:36535)
0620 18:35:08.563554 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:08.563556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:08.563565 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:08.563625 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:08.563626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:08.563635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:08.563685 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:08.564003 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.564511 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.565190 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.565389 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.565441 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.567824 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.65:50365: BlockingRecv error: recv got EOF from 127.10.109.65:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Jun 20 18:35:08 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:35:08 dist-test-slave-3jql krb5kdc[18806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.109.66@KRBTEST.COM for oryx/127.10.109.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:35:08.713678 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.707641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:36535 (local address 127.10.109.66:37157)
0620 18:35:08.707931 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:08.707949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:08.708097 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:08.708324 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:08.708325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:08.708564 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:08.708638 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.708640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.709516 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.709518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.710079 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.710081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.710565 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.710566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.710660 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.711368 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:08.711373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:08.713560 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.109.126:36535: Server oryx/127.10.109.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
W20260620 18:35:08.713706 19865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:08.707751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:37157 (local address 127.10.109.126:36535)
0620 18:35:08.707916 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:08.707919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:08.708064 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:35:08.708196 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:08.708198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:08.708215 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:08.708301 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:08.708731 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.709362 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.710177 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.710411 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.710487 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.713644 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.109.66:37157: BlockingRecv error: recv got EOF from 127.10.109.66:37157 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 18:35:08.743901 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19389
I20260620 18:35:08.748672 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19541
I20260620 18:35:08.752903 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19674
I20260620 18:35:08.757109 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19319
2026-06-20T18:35:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17453 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:08Z Disabled control of system clock
I20260620 18:35:08.784538 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44109
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:44109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:35:08.863677 19870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:08.863871 19870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:08.863891 19870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:08.864962 19870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:08.864996 19870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:08.865010 19870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:08.865021 19870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:08.865033 19870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:08.866305 19870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:44109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:08.866488 19870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:08.866714 19870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:08.871495 19876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:08.871568 19875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:08.871591 19878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:08.871841 19870 server_base.cc:1034] running on GCE node
I20260620 18:35:08.872133 19870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:08.872426 19870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:08.873638 19870 hybrid_clock.cc:648] HybridClock initialized: now 1781980508873572 us; error 78 us; skew 500 ppm
I20260620 18:35:08.875088 19870 webserver.cc:460] Webserver started at http://127.10.109.126:46173/ using document root <none> and password file <none>
I20260620 18:35:08.875345 19870 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:08.875417 19870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:08.875581 19870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:08.876638 19870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "fcdc7ab129c74996a4b2b39dd067c3fd"
format_stamp: "Formatted at 2026-06-20 18:35:08 on dist-test-slave-3jql"
server_key: "0810b6b073a57516570725cd8f70efb3"
server_key_iv: "c5e12129ee6786fd00ce06e1b6caa1ac"
server_key_version: "encryptionkey@0"
I20260620 18:35:08.877022 19870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "fcdc7ab129c74996a4b2b39dd067c3fd"
format_stamp: "Formatted at 2026-06-20 18:35:08 on dist-test-slave-3jql"
server_key: "0810b6b073a57516570725cd8f70efb3"
server_key_iv: "c5e12129ee6786fd00ce06e1b6caa1ac"
server_key_version: "encryptionkey@0"
I20260620 18:35:08.878995 19870 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:35:08.879882 19884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.880264 19870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:35:08.880396 19870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "fcdc7ab129c74996a4b2b39dd067c3fd"
format_stamp: "Formatted at 2026-06-20 18:35:08 on dist-test-slave-3jql"
server_key: "0810b6b073a57516570725cd8f70efb3"
server_key_iv: "c5e12129ee6786fd00ce06e1b6caa1ac"
server_key_version: "encryptionkey@0"
I20260620 18:35:08.880481 19870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:08.893911 19870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:08.894236 19870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:08.894402 19870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:08.898435 19870 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:44109
I20260620 18:35:08.898550 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:44109 every 8 connection(s)
I20260620 18:35:08.898885 19870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:08.899962 19937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:08.902232 19937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd: Bootstrap starting.
I20260620 18:35:08.902894 19937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:08.903192 19937 log.cc:824] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:08.903918 19937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd: No bootstrap required, opened a new log
I20260620 18:35:08.905627 19937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } }
I20260620 18:35:08.905774 19937 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:08.905797 19937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc7ab129c74996a4b2b39dd067c3fd, State: Initialized, Role: FOLLOWER
I20260620 18:35:08.905946 19937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } }
I20260620 18:35:08.906024 19937 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:08.906047 19937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:08.906100 19937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:08.906682 19937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } }
I20260620 18:35:08.906845 19937 leader_election.cc:302] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcdc7ab129c74996a4b2b39dd067c3fd; no voters:
I20260620 18:35:08.907030 19937 leader_election.cc:288] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:08.907105 19940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:08.907284 19940 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [term 1 LEADER]: Becoming Leader. State: Replica: fcdc7ab129c74996a4b2b39dd067c3fd, State: Running, Role: LEADER
I20260620 18:35:08.907286 19937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:08.907425 19940 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } }
I20260620 18:35:08.907819 19942 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcdc7ab129c74996a4b2b39dd067c3fd. Latest consensus state: current_term: 1 leader_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } } }
I20260620 18:35:08.907904 19941 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdc7ab129c74996a4b2b39dd067c3fd" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44109 } } }
I20260620 18:35:08.907979 19941 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:08.907909 19942 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:08.908222 19949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:08.908939 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19870
I20260620 18:35:08.908947 19949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:08.909030 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:35:08.909263 10677 external_mini_cluster.cc:1428] Setting key 223a9c9a598f5f3c7d2d0fe7a55ac599
I20260620 18:35:08.910599 19949 catalog_manager.cc:1347] Generated new cluster ID: 48730f6c383346c088c1cef7517ed12f
I20260620 18:35:08.910655 19949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:08.914342 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.909690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.10.109.126:44109)
0620 18:35:08.909877 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:08.909893 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:08.910169 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:35:08.910288 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:08.910294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:08.910345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:08.910485 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:08.910734 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.911620 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.912405 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.912817 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.912996 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.913669 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:08.913672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:08.913672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:08.913866 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:08.913899 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:08.913910 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:08.913984 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:08.914066 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 18:35:08.914144 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260620 18:35:08.919834 19949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:08.920291 19949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:08.925205 19949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcdc7ab129c74996a4b2b39dd067c3fd: Generated new TSK 0
I20260620 18:35:08.925379 19949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:08.927659 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--builtin_ntp_servers=127.10.109.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:35:09.012553 19961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.012760 19961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.012786 19961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.014082 19961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.014125 19961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.014153 19961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:35:09.015537 19961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.015765 19961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.016068 19961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.021935 19966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.021966 19969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.022006 19967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.022624 19961 server_base.cc:1034] running on GCE node
I20260620 18:35:09.022857 19961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.023144 19961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.024312 19961 hybrid_clock.cc:648] HybridClock initialized: now 1781980509024286 us; error 41 us; skew 500 ppm
I20260620 18:35:09.025681 19961 webserver.cc:460] Webserver started at http://127.10.109.65:46717/ using document root <none> and password file <none>
I20260620 18:35:09.025919 19961 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.025991 19961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.026127 19961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.027256 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "6380e43b0d6a4081a27f7fdc902b1c1b"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "511945abd1158f4771acbd7ecb5fab32"
server_key_iv: "1a2e79f8de94203b7fbb87f8873600b6"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.027601 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "6380e43b0d6a4081a27f7fdc902b1c1b"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "511945abd1158f4771acbd7ecb5fab32"
server_key_iv: "1a2e79f8de94203b7fbb87f8873600b6"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.028939 19961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:35:09.029747 19975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.029978 19961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:35:09.030079 19961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "6380e43b0d6a4081a27f7fdc902b1c1b"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "511945abd1158f4771acbd7ecb5fab32"
server_key_iv: "1a2e79f8de94203b7fbb87f8873600b6"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.030156 19961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.043484 19961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.043826 19961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.043996 19961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.044191 19961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.044484 19961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.044518 19961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.044543 19961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.044557 19961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.051386 19961 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:44631
I20260620 18:35:09.051460 20088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:44631 every 8 connection(s)
I20260620 18:35:09.051812 19961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:09.053781 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 19961
I20260620 18:35:09.053869 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:09.054098 10677 external_mini_cluster.cc:1428] Setting key 7b336f81fb3fa56d5b869754e1758118
I20260620 18:35:09.055169 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--builtin_ntp_servers=127.10.109.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:35:09.059031 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.053042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:53975 (local address 127.10.109.126:44109)
0620 18:35:09.053162 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.053164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.053525 (+ 361us) server_negotiation.cc:407] Connection header received
0620 18:35:09.054328 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.054330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.054346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.054423 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.055513 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.056215 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.057225 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.057545 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.057623 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.058605 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.058606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.058607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.058613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.058631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.058636 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.058699 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.058881 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.058914 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:35:09.059115 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.053020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44109 (local address 127.10.109.65:53975)
0620 18:35:09.053317 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.053371 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.054184 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.054515 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.054520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.054534 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.055405 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.055418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.056347 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.056349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.057154 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.057158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.057652 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.057653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.057787 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.058436 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.058437 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.058481 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.058505 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.058734 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.058779 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.058783 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.058855 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.058938 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":162,"threads_started":1}
I20260620 18:35:09.059652 20089 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44109
I20260620 18:35:09.059763 20089 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.059978 20089 heartbeater.cc:511] Master 127.10.109.126:44109 requested a full tablet report, sending...
I20260620 18:35:09.060523 19901 ts_manager.cc:194] Registered new tserver with Master: 6380e43b0d6a4081a27f7fdc902b1c1b (127.10.109.65:44631)
I20260620 18:35:09.061120 19901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.65:53975
W20260620 18:35:09.138302 20092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.138490 20092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.138510 20092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.139657 20092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.139696 20092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.139720 20092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:35:09.140998 20092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.141232 20092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.141523 20092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.146417 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.146521 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.146500 20092 server_base.cc:1034] running on GCE node
W20260620 18:35:09.146481 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.146937 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.147176 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.148356 20092 hybrid_clock.cc:648] HybridClock initialized: now 1781980509148345 us; error 29 us; skew 500 ppm
I20260620 18:35:09.149746 20092 webserver.cc:460] Webserver started at http://127.10.109.66:45219/ using document root <none> and password file <none>
I20260620 18:35:09.149993 20092 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.150069 20092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.150203 20092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.151266 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "455c27d465e2465c805f1a27385643fa"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "188f85537f691266281057eaef5fa488"
server_key_iv: "7b5af30c7dd05c864ad698c41f069871"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.151609 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "455c27d465e2465c805f1a27385643fa"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "188f85537f691266281057eaef5fa488"
server_key_iv: "7b5af30c7dd05c864ad698c41f069871"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.152829 20092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:35:09.153491 20106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.153656 20092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:09.153748 20092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "455c27d465e2465c805f1a27385643fa"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "188f85537f691266281057eaef5fa488"
server_key_iv: "7b5af30c7dd05c864ad698c41f069871"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.153831 20092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.163177 20092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.163434 20092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.163576 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.163770 20092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.164068 20092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.164129 20092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.164168 20092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.164197 20092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.169811 20092 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:46577
I20260620 18:35:09.169873 20219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:46577 every 8 connection(s)
I20260620 18:35:09.170179 20092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:09.175020 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.170882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:37005 (local address 127.10.109.126:44109)
0620 18:35:09.170926 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.170928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.171112 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:35:09.171574 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.171575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.171584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.171669 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.172298 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.172795 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.173727 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.173912 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.173969 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.174707 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.174709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.174709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.174713 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.174725 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.174729 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.174834 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.174944 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.174966 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:35:09.175216 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.170830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44109 (local address 127.10.109.66:37005)
0620 18:35:09.171002 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.171042 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.171491 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.171736 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.171740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.171748 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.172206 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.172209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.172876 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.172877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.173640 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.173643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.173989 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.173990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.174096 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.174539 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.174539 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.174572 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.174599 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.174839 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.174846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.174849 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.174932 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.175008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260620 18:35:09.175773 20220 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44109
I20260620 18:35:09.175853 20220 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.176055 20220 heartbeater.cc:511] Master 127.10.109.126:44109 requested a full tablet report, sending...
I20260620 18:35:09.176422 19901 ts_manager.cc:194] Registered new tserver with Master: 455c27d465e2465c805f1a27385643fa (127.10.109.66:46577)
I20260620 18:35:09.176739 19901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.66:37005
I20260620 18:35:09.180095 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20092
I20260620 18:35:09.180169 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:09.180326 10677 external_mini_cluster.cc:1428] Setting key 32a5af795543384c023a7dc0c5758ea2
I20260620 18:35:09.181118 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--builtin_ntp_servers=127.10.109.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:35:09.261586 20223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.261786 20223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.261806 20223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.262991 20223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.263029 20223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.263053 20223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:35:09.264358 20223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.264592 20223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.264871 20223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.269981 20229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.269987 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.270164 20223 server_base.cc:1034] running on GCE node
W20260620 18:35:09.270045 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.270435 20223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.270687 20223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.271845 20223 hybrid_clock.cc:648] HybridClock initialized: now 1781980509271837 us; error 30 us; skew 500 ppm
I20260620 18:35:09.273043 20223 webserver.cc:460] Webserver started at http://127.10.109.67:39149/ using document root <none> and password file <none>
I20260620 18:35:09.273316 20223 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.273391 20223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.273530 20223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.274502 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "55433c04af5746be9281b2dff76104d1"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "b55fc9f9b7b506bf5500b32309d6dd07"
server_key_iv: "508a175c36ea5cdf6167cea55e1014d2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.274911 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "55433c04af5746be9281b2dff76104d1"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "b55fc9f9b7b506bf5500b32309d6dd07"
server_key_iv: "508a175c36ea5cdf6167cea55e1014d2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.276232 20223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:35:09.276954 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.277144 20223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:35:09.277248 20223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "55433c04af5746be9281b2dff76104d1"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "b55fc9f9b7b506bf5500b32309d6dd07"
server_key_iv: "508a175c36ea5cdf6167cea55e1014d2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.277335 20223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.293437 20223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.293716 20223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.293864 20223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.294064 20223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.294363 20223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.294416 20223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.294457 20223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.294484 20223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.300065 20223 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:36935
I20260620 18:35:09.300105 20350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:36935 every 8 connection(s)
I20260620 18:35:09.300422 20223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:09.305601 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.301283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:59533 (local address 127.10.109.126:44109)
0620 18:35:09.301358 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.301360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.301373 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:35:09.301894 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.301896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.301908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.301968 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.302659 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.303173 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.304143 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.304386 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.304462 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.305321 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.305323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.305323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.305328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.305343 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.305348 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.305396 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.305485 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.305517 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:35:09.305800 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.301059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44109 (local address 127.10.109.67:59533)
0620 18:35:09.301248 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.301286 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.301786 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.302061 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.302065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.302073 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.302566 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.302570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.303271 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.303272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.304030 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.304032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.304478 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.304480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.304608 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.305167 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.305168 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.305194 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.305215 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.305395 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.305400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.305403 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.305487 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.305582 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260620 18:35:09.305790 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20223
I20260620 18:35:09.305907 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:09.306121 10677 external_mini_cluster.cc:1428] Setting key 9f75e3d39d9f2c957f2a990923fcf72d
I20260620 18:35:09.306597 20351 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44109
I20260620 18:35:09.306720 20351 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.306915 20351 heartbeater.cc:511] Master 127.10.109.126:44109 requested a full tablet report, sending...
I20260620 18:35:09.307247 19901 ts_manager.cc:194] Registered new tserver with Master: 55433c04af5746be9281b2dff76104d1 (127.10.109.67:36935)
I20260620 18:35:09.307619 19901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.67:59533
I20260620 18:35:09.318065 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:35:09.320144 19959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:09.319601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.10.109.126:44109)
0620 18:35:09.319670 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.319672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.319681 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:09.319784 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.319784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.319795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.319849 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.320073 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51782: BlockingRecv error: recv got EOF from 127.0.0.1:51782 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:35:09.320237 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19961
I20260620 18:35:09.325129 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20092
I20260620 18:35:09.329245 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20223
I20260620 18:35:09.333626 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 19870
2026-06-20T18:35:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (575 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:09Z Disabled control of system clock
I20260620 18:35:09.360132 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43429
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:35:09.440038 20362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.440240 20362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.440261 20362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.441309 20362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:09.441340 20362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.441354 20362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.441365 20362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:09.441377 20362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:09.442605 20362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.442852 20362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.443086 20362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.447973 20370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.448071 20368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.448074 20367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.448235 20362 server_base.cc:1034] running on GCE node
I20260620 18:35:09.448560 20362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.448872 20362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.450064 20362 hybrid_clock.cc:648] HybridClock initialized: now 1781980509450053 us; error 45 us; skew 500 ppm
I20260620 18:35:09.451364 20362 webserver.cc:460] Webserver started at http://127.10.109.126:36115/ using document root <none> and password file <none>
I20260620 18:35:09.451613 20362 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.451689 20362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.451836 20362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.452867 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "be45dc61f2ad4853b95368869743d1da"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "9cda67964485a1e2632967f5625ac03b"
server_key_iv: "6f49bb9b3debb5bd3ff7ce284860c321"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.453222 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "be45dc61f2ad4853b95368869743d1da"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "9cda67964485a1e2632967f5625ac03b"
server_key_iv: "6f49bb9b3debb5bd3ff7ce284860c321"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.454555 20362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:35:09.455370 20376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.455544 20362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.455641 20362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "be45dc61f2ad4853b95368869743d1da"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "9cda67964485a1e2632967f5625ac03b"
server_key_iv: "6f49bb9b3debb5bd3ff7ce284860c321"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.455725 20362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.458875 20362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.459120 20362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.459256 20362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.462888 20362 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:43429
I20260620 18:35:09.462916 20428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:43429 every 8 connection(s)
I20260620 18:35:09.463236 20362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:09.463882 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:09.464128 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20362
I20260620 18:35:09.464231 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:35:09.464428 10677 external_mini_cluster.cc:1428] Setting key b6f04dbc6eaf8bc849034ddf4870ea11
I20260620 18:35:09.466853 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.464973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.10.109.126:43429)
0620 18:35:09.465163 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.465169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.465191 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:35:09.465556 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.465561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.465609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.465746 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.466395 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.466398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.466398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.466402 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.466446 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.466459 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.466537 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.466604 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.466657 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260620 18:35:09.467707 20429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da: Bootstrap starting.
I20260620 18:35:09.468273 20429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:09.468547 20429 log.cc:824] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:09.469214 20429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da: No bootstrap required, opened a new log
I20260620 18:35:09.470543 20429 raft_consensus.cc:348] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } }
I20260620 18:35:09.470733 20429 raft_consensus.cc:374] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:09.470840 20429 raft_consensus.cc:729] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be45dc61f2ad4853b95368869743d1da, State: Initialized, Role: FOLLOWER
I20260620 18:35:09.470963 20429 consensus_queue.cc:260] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } }
I20260620 18:35:09.471053 20429 raft_consensus.cc:388] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:09.471096 20429 raft_consensus.cc:482] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:09.471143 20429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:09.471628 20429 raft_consensus.cc:504] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } }
I20260620 18:35:09.471747 20429 leader_election.cc:302] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be45dc61f2ad4853b95368869743d1da; no voters:
I20260620 18:35:09.471894 20429 leader_election.cc:288] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:09.471938 20434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:09.472085 20434 raft_consensus.cc:686] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [term 1 LEADER]: Becoming Leader. State: Replica: be45dc61f2ad4853b95368869743d1da, State: Running, Role: LEADER
I20260620 18:35:09.472173 20429 sys_catalog.cc:564] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:09.472189 20434 consensus_queue.cc:237] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } }
I20260620 18:35:09.472515 20434 sys_catalog.cc:455] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [sys.catalog]: SysCatalogTable state changed. Reason: New leader be45dc61f2ad4853b95368869743d1da. Latest consensus state: current_term: 1 leader_uuid: "be45dc61f2ad4853b95368869743d1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } } }
I20260620 18:35:09.472594 20434 sys_catalog.cc:458] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:09.472527 20435 sys_catalog.cc:455] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be45dc61f2ad4853b95368869743d1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be45dc61f2ad4853b95368869743d1da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43429 } } }
I20260620 18:35:09.472749 20435 sys_catalog.cc:458] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:09.472878 20439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:09.473416 20439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:09.476327 20439 catalog_manager.cc:1347] Generated new cluster ID: b738991b110a45f882654c268900488f
I20260620 18:35:09.476380 20439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:09.486845 20439 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:09.486992 20439 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:09.491143 20439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be45dc61f2ad4853b95368869743d1da: Generated new TSK 0
I20260620 18:35:09.491291 20439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:09.501618 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--builtin_ntp_servers=127.10.109.84:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:35:09.580849 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.581034 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.581056 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.582193 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.582231 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.582257 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:35:09.583567 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.583802 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.584084 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.589802 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.589857 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.589795 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.590169 20453 server_base.cc:1034] running on GCE node
I20260620 18:35:09.590373 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.590634 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.591809 20453 hybrid_clock.cc:648] HybridClock initialized: now 1781980509591793 us; error 33 us; skew 500 ppm
I20260620 18:35:09.593040 20453 webserver.cc:460] Webserver started at http://127.10.109.65:43287/ using document root <none> and password file <none>
I20260620 18:35:09.593298 20453 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.593370 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.593502 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.594547 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "eda38f0f2a1f43d4922765654bb9876d"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "ce7a5ae626f0f71f535f234b4ea87a7c"
server_key_iv: "47367de2596e46a4e19f132968aa5cd2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.594928 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "eda38f0f2a1f43d4922765654bb9876d"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "ce7a5ae626f0f71f535f234b4ea87a7c"
server_key_iv: "47367de2596e46a4e19f132968aa5cd2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.596225 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:35:09.596897 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.597070 20453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:35:09.597162 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "eda38f0f2a1f43d4922765654bb9876d"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "ce7a5ae626f0f71f535f234b4ea87a7c"
server_key_iv: "47367de2596e46a4e19f132968aa5cd2"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.597244 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.600504 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.600759 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.600909 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.601143 20453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.601439 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.601500 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.601538 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.601565 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.607498 20453 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:46227
I20260620 18:35:09.607533 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:46227 every 8 connection(s)
I20260620 18:35:09.607837 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:09.610653 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.608590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:40621 (local address 127.10.109.126:43429)
0620 18:35:09.608633 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.608634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.608879 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:35:09.609314 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.609315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.609320 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.609371 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.610474 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.610475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.610475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.610478 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.610487 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.610489 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.610527 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.610584 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.610594 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:35:09.610878 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.608590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43429 (local address 127.10.109.65:40621)
0620 18:35:09.608771 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.608800 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.609236 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.609418 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.609420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.609428 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.610357 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.610358 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.610385 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.610410 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.610528 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.610530 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.610532 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.610588 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.610631 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260620 18:35:09.611433 20581 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43429
I20260620 18:35:09.611537 20581 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.611697 20581 heartbeater.cc:511] Master 127.10.109.126:43429 requested a full tablet report, sending...
I20260620 18:35:09.612187 20393 ts_manager.cc:194] Registered new tserver with Master: eda38f0f2a1f43d4922765654bb9876d (127.10.109.65:46227)
I20260620 18:35:09.616169 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20453
I20260620 18:35:09.616252 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:09.616402 10677 external_mini_cluster.cc:1428] Setting key e45070cc0cdadd357975096164825056
I20260620 18:35:09.617210 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--builtin_ntp_servers=127.10.109.84:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:35:09.695940 20584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.696174 20584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.696201 20584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.697359 20584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.697397 20584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.697422 20584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:35:09.698629 20584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.698858 20584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.699084 20584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.703958 20589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.704077 20584 server_base.cc:1034] running on GCE node
W20260620 18:35:09.704003 20590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.703958 20592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.704416 20584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.704595 20584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.705763 20584 hybrid_clock.cc:648] HybridClock initialized: now 1781980509705739 us; error 40 us; skew 500 ppm
I20260620 18:35:09.707082 20584 webserver.cc:460] Webserver started at http://127.10.109.66:41449/ using document root <none> and password file <none>
I20260620 18:35:09.707271 20584 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.707312 20584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.707396 20584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.708326 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "868a1124a5174594bf822f3bfcf303ae"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "c107a50d25198848787557f6e11e81d2"
server_key_iv: "f2b9a4e9cc2ea3dc0250901faee8f103"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.708601 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "868a1124a5174594bf822f3bfcf303ae"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "c107a50d25198848787557f6e11e81d2"
server_key_iv: "f2b9a4e9cc2ea3dc0250901faee8f103"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.709789 20584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:35:09.710503 20598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.710695 20584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:35:09.710813 20584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "868a1124a5174594bf822f3bfcf303ae"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "c107a50d25198848787557f6e11e81d2"
server_key_iv: "f2b9a4e9cc2ea3dc0250901faee8f103"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.710870 20584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.714665 20584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.714962 20584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.715075 20584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.715242 20584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.715504 20584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.715591 20584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.715615 20584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.715628 20584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.721911 20584 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:38475
I20260620 18:35:09.721974 20711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:38475 every 8 connection(s)
I20260620 18:35:09.722301 20584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:09.725253 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.723120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:50441 (local address 127.10.109.126:43429)
0620 18:35:09.723179 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.723181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.723391 (+ 210us) server_negotiation.cc:407] Connection header received
0620 18:35:09.723854 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.723855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.723861 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.723921 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.725062 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.725064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.725064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.725068 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.725082 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.725086 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.725133 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.725192 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.725203 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:35:09.725476 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.723077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43429 (local address 127.10.109.66:50441)
0620 18:35:09.723256 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.723299 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.723763 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.723970 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.723973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.723981 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.724917 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.724918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.724946 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.724973 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.725134 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.725136 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.725138 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.725198 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.725255 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 18:35:09.725986 20712 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43429
I20260620 18:35:09.726060 20712 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.726209 20712 heartbeater.cc:511] Master 127.10.109.126:43429 requested a full tablet report, sending...
I20260620 18:35:09.726640 20393 ts_manager.cc:194] Registered new tserver with Master: 868a1124a5174594bf822f3bfcf303ae (127.10.109.66:38475)
I20260620 18:35:09.731726 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20584
I20260620 18:35:09.731858 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:09.732046 10677 external_mini_cluster.cc:1428] Setting key eb2d8f270f33a262525f7ddccb34abf8
I20260620 18:35:09.732905 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--builtin_ntp_servers=127.10.109.84:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:35:09.812332 20715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.812530 20715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.812553 20715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.813753 20715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.813788 20715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.813813 20715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:35:09.815253 20715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:09.815495 20715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.815778 20715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.820924 20721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.820951 20723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.820957 20720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.821259 20715 server_base.cc:1034] running on GCE node
I20260620 18:35:09.821453 20715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.821637 20715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.822814 20715 hybrid_clock.cc:648] HybridClock initialized: now 1781980509822801 us; error 31 us; skew 500 ppm
I20260620 18:35:09.824019 20715 webserver.cc:460] Webserver started at http://127.10.109.67:35117/ using document root <none> and password file <none>
I20260620 18:35:09.824251 20715 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.824298 20715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:09.824471 20715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:09.825503 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "03ab65863cb3446aaf879193dfac262a"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "75f0cbfcd0d0bb42447890d9dcc626ef"
server_key_iv: "2a9e64b6b29b99ba5e46e423c63adc06"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.825874 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "03ab65863cb3446aaf879193dfac262a"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "75f0cbfcd0d0bb42447890d9dcc626ef"
server_key_iv: "2a9e64b6b29b99ba5e46e423c63adc06"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.827318 20715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:35:09.828047 20729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.828261 20715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:35:09.828329 20715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "03ab65863cb3446aaf879193dfac262a"
format_stamp: "Formatted at 2026-06-20 18:35:09 on dist-test-slave-3jql"
server_key: "75f0cbfcd0d0bb42447890d9dcc626ef"
server_key_iv: "2a9e64b6b29b99ba5e46e423c63adc06"
server_key_version: "encryptionkey@0"
I20260620 18:35:09.828379 20715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.831501 20715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.831705 20715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.831796 20715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.831934 20715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.832150 20715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.832185 20715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.832204 20715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.832216 20715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.838141 20715 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:43193
I20260620 18:35:09.838179 20842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:43193 every 8 connection(s)
I20260620 18:35:09.838546 20715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:09.841636 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.839328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:52707 (local address 127.10.109.126:43429)
0620 18:35:09.839383 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.839385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.839611 (+ 226us) server_negotiation.cc:407] Connection header received
0620 18:35:09.840313 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.840314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.840319 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.840376 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.841439 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.841440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.841440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:09.841444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.841458 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:09.841462 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.841510 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.841572 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.841584 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:35:09.841789 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.839304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43429 (local address 127.10.109.67:52707)
0620 18:35:09.839490 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.839527 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.840190 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.840431 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.840434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.840441 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.841291 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:35:09.841291 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.841318 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:35:09.841343 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.841508 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.841511 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.841513 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.841562 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.841606 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 18:35:09.842363 20843 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43429
I20260620 18:35:09.842442 20843 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.842744 20843 heartbeater.cc:511] Master 127.10.109.126:43429 requested a full tablet report, sending...
I20260620 18:35:09.843313 20393 ts_manager.cc:194] Registered new tserver with Master: 03ab65863cb3446aaf879193dfac262a (127.10.109.67:43193)
I20260620 18:35:09.847625 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20715
I20260620 18:35:09.847707 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:09.847908 10677 external_mini_cluster.cc:1428] Setting key 5fdae1d6fafa91686e52baf3f6ec0cc5
I20260620 18:35:09.849016 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:35:09.851231 20432 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:09.850524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.10.109.126:43429)
0620 18:35:09.850605 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.850608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.850676 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:35:09.850810 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.850811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.850825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.850894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.851108 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59452: BlockingRecv error: recv got EOF from 127.0.0.1:59452 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:35:09.851327 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20453
I20260620 18:35:09.855676 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20584
I20260620 18:35:09.860313 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20715
I20260620 18:35:09.864302 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20362
2026-06-20T18:35:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:09 dist-test-slave-3jql krb5kdc[20853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:09 dist-test-slave-3jql krb5kdc[20853](info): set up 2 sockets
Jun 20 18:35:09 dist-test-slave-3jql krb5kdc[20853](info): commencing operation
krb5kdc: starting...
W20260620 18:35:10.936882 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:35:11.058074 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33021
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:11.136492 20866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.136683 20866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.136704 20866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:11.137739 20866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:11.137773 20866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.137786 20866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.137800 20866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:11.137812 20866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:11.139174 20866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:11.139344 20866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.139492 20866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.144085 20872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.144070 20871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.144070 20874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.144337 20866 server_base.cc:1034] running on GCE node
I20260620 18:35:11.144541 20866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.144817 20866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.145979 20866 hybrid_clock.cc:648] HybridClock initialized: now 1781980511145962 us; error 26 us; skew 500 ppm
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.148350 20866 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:35:11.148715 20866 webserver.cc:460] Webserver started at http://127.10.109.126:39099/ using document root <none> and password file <none>
I20260620 18:35:11.148880 20866 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.148919 20866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.149000 20866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.149915 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7049b79c3eae4c5dbd80653fd962fe16"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "cd774f562d8685336ce17f771390fd4f"
server_key_iv: "20c4135f221130d4614e5e069f50fb79"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.150200 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7049b79c3eae4c5dbd80653fd962fe16"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "cd774f562d8685336ce17f771390fd4f"
server_key_iv: "20c4135f221130d4614e5e069f50fb79"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.151485 20866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:35:11.152207 20881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.152406 20866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:11.152485 20866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7049b79c3eae4c5dbd80653fd962fe16"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "cd774f562d8685336ce17f771390fd4f"
server_key_iv: "20c4135f221130d4614e5e069f50fb79"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.152571 20866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.164992 20866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.167125 20866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.167265 20866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.170994 20866 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:33021
I20260620 18:35:11.171057 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:33021 every 8 connection(s)
I20260620 18:35:11.171368 20866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:11.172058 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20866
I20260620 18:35:11.172185 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:35:11.172384 10677 external_mini_cluster.cc:1428] Setting key e75d657c07acaf1946cb555d39bad765
I20260620 18:35:11.172566 20934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.175413 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: Bootstrap starting.
I20260620 18:35:11.176023 20934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:11.176317 20934 log.cc:824] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:11.176946 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: No bootstrap required, opened a new log
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:11.178781 20934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } }
I20260620 18:35:11.178937 20934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:11.178982 20934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7049b79c3eae4c5dbd80653fd962fe16, State: Initialized, Role: FOLLOWER
I20260620 18:35:11.179106 20934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } }
I20260620 18:35:11.179212 20934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:11.179268 20934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:11.179318 20934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:11.179816 20934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } }
I20260620 18:35:11.179934 20934 leader_election.cc:302] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7049b79c3eae4c5dbd80653fd962fe16; no voters:
I20260620 18:35:11.180102 20934 leader_election.cc:288] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:11.180145 20939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:11.180325 20939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [term 1 LEADER]: Becoming Leader. State: Replica: 7049b79c3eae4c5dbd80653fd962fe16, State: Running, Role: LEADER
I20260620 18:35:11.180361 20934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:11.180480 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } }
I20260620 18:35:11.180838 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7049b79c3eae4c5dbd80653fd962fe16. Latest consensus state: current_term: 1 leader_uuid: "7049b79c3eae4c5dbd80653fd962fe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } } }
I20260620 18:35:11.180853 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7049b79c3eae4c5dbd80653fd962fe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7049b79c3eae4c5dbd80653fd962fe16" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33021 } } }
I20260620 18:35:11.181011 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:11.181068 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:11.181243 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:11.181988 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:11.183009 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.172872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.10.109.126:33021)
0620 18:35:11.173058 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.173073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.173104 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:35:11.173623 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.173630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.173672 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.173808 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.174390 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.175299 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.176085 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.176559 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.176686 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.178972 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.178975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.178976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.178988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.180996 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.181431 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.181433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.181434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.181528 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.181758 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.181758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.181759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.182045 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.182612 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.182682 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.182834 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260620 18:35:11.183861 20948 catalog_manager.cc:1347] Generated new cluster ID: 732f59174030410491a57989a68a27eb
I20260620 18:35:11.183902 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:11.201152 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:11.201628 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:11.208242 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: Generated new TSK 0
I20260620 18:35:11.208403 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:35:11.235888 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:11.321065 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.321261 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.321285 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:11.322396 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.322441 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.322467 20960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:35:11.323881 20960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:11.324107 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.324339 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.329216 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.329200 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.329197 20960 server_base.cc:1034] running on GCE node
W20260620 18:35:11.329250 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.329674 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.329866 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.330994 20960 hybrid_clock.cc:648] HybridClock initialized: now 1781980511330981 us; error 30 us; skew 500 ppm
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.333369 20960 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:35:11.333820 20960 webserver.cc:460] Webserver started at http://127.10.109.65:42491/ using document root <none> and password file <none>
I20260620 18:35:11.334043 20960 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.334086 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.334239 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.335278 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "991c381955f92740a526f0dcfa0a5ced"
server_key_iv: "13c38fba3630318e63117c0502cc50dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.335615 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "991c381955f92740a526f0dcfa0a5ced"
server_key_iv: "13c38fba3630318e63117c0502cc50dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.336905 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:35:11.337617 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.337815 20960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:11.337908 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "991c381955f92740a526f0dcfa0a5ced"
server_key_iv: "13c38fba3630318e63117c0502cc50dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.337977 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.352905 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.354602 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.354790 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.355017 20960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:11.355342 20960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:11.355401 20960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.355435 20960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:11.355461 20960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.362464 20960 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:41363
I20260620 18:35:11.362881 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:11.362938 21088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:41363 every 8 connection(s)
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:11.370698 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 20960
I20260620 18:35:11.370823 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:11.371037 10677 external_mini_cluster.cc:1428] Setting key b33612337fd30d6a8f0cdaf6d02076c7
I20260620 18:35:11.372444 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.364252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:57025 (local address 127.10.109.126:33021)
0620 18:35:11.364297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.364299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.364310 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:35:11.364792 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.364794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.364807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.364862 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.365894 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.366448 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.367485 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.367699 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.367779 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.369686 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.369687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.369687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.369695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.371109 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.371537 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.371538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.371538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.371598 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.371863 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.371864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.371864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.371960 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.372018 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.372320 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.372381 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:35:11.372746 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.363808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:33021 (local address 127.10.109.65:57025)
0620 18:35:11.363991 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.364031 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.364649 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.364933 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.364937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.365498 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.365773 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.365779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.366546 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.366546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.367383 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.367386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.367789 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.367790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.367905 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.368303 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.368310 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.369596 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.371202 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.371209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.371211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.371474 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.371666 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.371667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.371667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.371797 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.372025 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.372031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.372187 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.372304 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.372473 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260620 18:35:11.373209 21089 heartbeater.cc:347] Connected to a master server at 127.10.109.126:33021
I20260620 18:35:11.373296 21089 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:11.373457 21089 heartbeater.cc:511] Master 127.10.109.126:33021 requested a full tablet report, sending...
I20260620 18:35:11.373942 20898 ts_manager.cc:194] Registered new tserver with Master: 0d8aca8b97ec4e3eb613e09d8ef6240d (127.10.109.65:41363)
I20260620 18:35:11.374864 20898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:57025
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:35:11.395298 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:11.474503 21094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.474704 21094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.474726 21094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:11.475873 21094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.475911 21094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.475937 21094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:35:11.477341 21094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:11.477568 21094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.477797 21094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.482658 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.482707 21100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.482674 21099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.482977 21094 server_base.cc:1034] running on GCE node
I20260620 18:35:11.483227 21094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.483420 21094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.484550 21094 hybrid_clock.cc:648] HybridClock initialized: now 1781980511484538 us; error 29 us; skew 500 ppm
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.486937 21094 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:35:11.487300 21094 webserver.cc:460] Webserver started at http://127.10.109.66:35799/ using document root <none> and password file <none>
I20260620 18:35:11.487468 21094 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.487507 21094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.487593 21094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.488533 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "61333d01e477499fb252b0c118cd751a"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "ef1494c0a295e2ba492989d76bf0207c"
server_key_iv: "b036d00968a91b441e30170f0735039e"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.488830 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "61333d01e477499fb252b0c118cd751a"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "ef1494c0a295e2ba492989d76bf0207c"
server_key_iv: "b036d00968a91b441e30170f0735039e"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.490108 21094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:35:11.490921 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.491081 21094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:35:11.491176 21094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "61333d01e477499fb252b0c118cd751a"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "ef1494c0a295e2ba492989d76bf0207c"
server_key_iv: "b036d00968a91b441e30170f0735039e"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.491274 21094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.508504 21094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.510622 21094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.510833 21094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.511071 21094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:11.511384 21094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:11.511436 21094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.511476 21094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:11.511504 21094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.517266 21094 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:33871
I20260620 18:35:11.517308 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:33871 every 8 connection(s)
I20260620 18:35:11.517640 21094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:11.519506 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 21094
I20260620 18:35:11.519615 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:11.519913 10677 external_mini_cluster.cc:1428] Setting key c53ebeea88bfc8906303a3fd41da0a56
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:11.528841 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.518397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:39485 (local address 127.10.109.126:33021)
0620 18:35:11.518463 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.518465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.518669 (+ 204us) server_negotiation.cc:407] Connection header received
0620 18:35:11.519204 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.519205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.519215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.519277 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.520759 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.521260 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.522273 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.522475 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.522645 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.525753 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.525755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.525755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.525766 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.527424 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.527858 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.527859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.527860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.527919 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.528170 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.528171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.528171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.528291 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.528364 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.528658 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.528740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:35:11.529059 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.518347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:33021 (local address 127.10.109.66:39485)
0620 18:35:11.518503 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.518543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.519093 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.519370 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.519373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.520072 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.520440 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.520446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.521359 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.521359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.522167 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.522169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.522705 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.522708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.522996 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.523712 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.523725 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.525648 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.527537 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.527544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.527546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.527782 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.527992 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.527992 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.527993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.528103 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.528357 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.528363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.528522 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.528635 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.528805 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:35:11.529587 21223 heartbeater.cc:347] Connected to a master server at 127.10.109.126:33021
I20260620 18:35:11.529695 21223 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:11.529886 21223 heartbeater.cc:511] Master 127.10.109.126:33021 requested a full tablet report, sending...
I20260620 18:35:11.530298 20898 ts_manager.cc:194] Registered new tserver with Master: 61333d01e477499fb252b0c118cd751a (127.10.109.66:33871)
I20260620 18:35:11.530683 20898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:39485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:35:11.549995 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:11.630513 21228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:11.630823 21228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:11.630857 21228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:11.632684 21228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:11.632745 21228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:11.632786 21228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:35:11.635248 21228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:11.635484 21228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:11.635748 21228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:11.641220 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.641252 21228 server_base.cc:1034] running on GCE node
W20260620 18:35:11.641258 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:11.641227 21236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:11.641554 21228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:11.641744 21228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:11.642877 21228 hybrid_clock.cc:648] HybridClock initialized: now 1781980511642865 us; error 26 us; skew 500 ppm
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:11.645301 21228 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:35:11.645745 21228 webserver.cc:460] Webserver started at http://127.10.109.67:37199/ using document root <none> and password file <none>
I20260620 18:35:11.645962 21228 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:11.646024 21228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:11.646176 21228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:11.647272 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "16efac7818ae4968934071e28121e808"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "9ea304b25e3039983b4bdbb7832a31a1"
server_key_iv: "c3827bb05891013caf1808fec3001ba5"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.647619 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "16efac7818ae4968934071e28121e808"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "9ea304b25e3039983b4bdbb7832a31a1"
server_key_iv: "c3827bb05891013caf1808fec3001ba5"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.648902 21228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:35:11.649590 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.649763 21228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:11.649854 21228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "16efac7818ae4968934071e28121e808"
format_stamp: "Formatted at 2026-06-20 18:35:11 on dist-test-slave-3jql"
server_key: "9ea304b25e3039983b4bdbb7832a31a1"
server_key_iv: "c3827bb05891013caf1808fec3001ba5"
server_key_version: "encryptionkey@0"
I20260620 18:35:11.649927 21228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:11.670646 21228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:11.673149 21228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:11.673316 21228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:11.673554 21228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:11.673866 21228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:11.673915 21228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.673957 21228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:11.673985 21228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:11.680142 21228 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:41843
I20260620 18:35:11.680188 21356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:41843 every 8 connection(s)
I20260620 18:35:11.680508 21228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:11.685871 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 21228
I20260620 18:35:11.685972 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:11.686162 10677 external_mini_cluster.cc:1428] Setting key b4892e98741a13b21161f19da9001b8b
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:11.690068 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.681336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:43933 (local address 127.10.109.126:33021)
0620 18:35:11.681392 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.681395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.681612 (+ 217us) server_negotiation.cc:407] Connection header received
0620 18:35:11.682225 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.682226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.682239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.682294 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.683317 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.683845 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.684872 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.685079 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.685157 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.687610 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.687611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.687612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.687619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.688762 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.689160 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.689161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.689162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.689226 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.689463 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.689463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.689464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.689572 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.689646 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.689900 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.689978 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:35:11.690256 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.681350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:33021 (local address 127.10.109.67:43933)
0620 18:35:11.681502 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.681536 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.682114 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.682359 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.682363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.682999 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:11.683222 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.683226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.683950 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.683951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.684776 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.684779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.685167 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.685168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.685290 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.685777 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:11.685784 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:11.687525 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:11.688856 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.688862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.688864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.689075 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.689290 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:11.689290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:11.689291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:11.689399 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:11.689635 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:11.689641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:11.689777 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.689879 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.690026 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260620 18:35:11.690637 21357 heartbeater.cc:347] Connected to a master server at 127.10.109.126:33021
I20260620 18:35:11.690725 21357 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:11.690908 21357 heartbeater.cc:511] Master 127.10.109.126:33021 requested a full tablet report, sending...
I20260620 18:35:11.691226 20898 ts_manager.cc:194] Registered new tserver with Master: 16efac7818ae4968934071e28121e808 (127.10.109.67:41843)
I20260620 18:35:11.691536 20898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:43933
I20260620 18:35:11.698247 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:11.705298 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.699720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.10.109.126:33021)
0620 18:35:11.699799 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.699801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.699886 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:35:11.699907 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.699908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.699916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.699964 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:11.700479 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.700954 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.701660 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.701855 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.701918 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.703124 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:11.703125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:11.703125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:11.703131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:11.704390 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.704706 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.704707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.704707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.704756 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:11.704913 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:11.704913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:11.704914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:11.705037 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:11.705108 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.705185 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.705239 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:35:11.706838 20898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:11.708570 20898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:11.714959 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.710820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:41843 (local address 127.0.0.1:52540)
0620 18:35:11.710980 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.711017 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.711081 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.711539 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.711540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.711543 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:11.711619 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.711621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.712834 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.712835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.713898 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.713899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.714576 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.714670 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.714689 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.714792 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.714855 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
I20260620 18:35:11.715003 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.711081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.10.109.66:33871)
0620 18:35:11.711255 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.711259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.711821 (+ 562us) server_negotiation.cc:407] Connection header received
0620 18:35:11.711888 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.711890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.711906 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.711984 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:11.712095 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.713038 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714139 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.714626 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714703 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.714784 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.714878 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.714925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260620 18:35:11.715210 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.710847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:33871 (local address 127.0.0.1:47592)
0620 18:35:11.711767 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.711777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.711826 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.711992 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.711993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.711995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:11.712041 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.712042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.713131 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.713132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.714061 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.714064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.714719 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.714803 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.714807 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.714863 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.714906 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":71,"threads_started":1}
I20260620 18:35:11.715430 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.711068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.10.109.67:41843)
0620 18:35:11.711380 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.711383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.711390 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:35:11.711442 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.711445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.711459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.711540 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:11.711680 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.712735 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.713978 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.714500 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714554 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.714591 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.714947 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.714990 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":73,"threads_started":1}
I20260620 18:35:11.715788 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.710821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41363 (local address 127.0.0.1:41660)
0620 18:35:11.711221 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.711231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.711287 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.711448 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.711450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.711453 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:11.711525 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.711527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.712620 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.712621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.713896 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.713899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.715542 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.715543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.715611 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.715614 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.715681 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.715721 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260620 18:35:11.715934 21157 tablet_service.cc:1467] Processing CreateTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d (DEFAULT_TABLE table=test-table [id=b1d2049b42b54e0d9597ac4b1bd3cd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:11.716092 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.710978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.10.109.65:41363)
0620 18:35:11.711172 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.711178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.711298 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:35:11.711339 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.711341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.711353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.711421 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:11.711613 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.712509 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.714633 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.715458 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.715543 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.715597 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.715684 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.715839 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260620 18:35:11.716280 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cae3f621c34134bcc38e4e7abb2d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.716612 21021 tablet_service.cc:1467] Processing CreateTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d (DEFAULT_TABLE table=test-table [id=b1d2049b42b54e0d9597ac4b1bd3cd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:11.716884 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cae3f621c34134bcc38e4e7abb2d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.719246 21374 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Bootstrap starting.
I20260620 18:35:11.719352 21291 tablet_service.cc:1467] Processing CreateTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d (DEFAULT_TABLE table=test-table [id=b1d2049b42b54e0d9597ac4b1bd3cd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:11.719496 21375 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Bootstrap starting.
I20260620 18:35:11.719682 21291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cae3f621c34134bcc38e4e7abb2d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:11.719995 21374 tablet_bootstrap.cc:654] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:11.720232 21375 tablet_bootstrap.cc:654] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:11.720499 21374 log.cc:824] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:11.720772 21375 log.cc:824] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:11.721243 21374 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: No bootstrap required, opened a new log
I20260620 18:35:11.721328 21374 ts_tablet_manager.cc:1397] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:35:11.721586 21375 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: No bootstrap required, opened a new log
I20260620 18:35:11.721658 21375 ts_tablet_manager.cc:1397] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:11.722945 21374 raft_consensus.cc:348] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.723099 21374 raft_consensus.cc:374] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:11.723142 21374 raft_consensus.cc:729] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61333d01e477499fb252b0c118cd751a, State: Initialized, Role: FOLLOWER
I20260620 18:35:11.723251 21374 consensus_queue.cc:260] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.723312 21378 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Bootstrap starting.
I20260620 18:35:11.723527 21374 ts_tablet_manager.cc:1428] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:35:11.723691 21223 heartbeater.cc:503] Master 127.10.109.126:33021 was elected leader, sending a full tablet report...
I20260620 18:35:11.724049 21378 tablet_bootstrap.cc:654] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:11.724076 21375 raft_consensus.cc:348] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.724236 21375 raft_consensus.cc:374] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:11.724274 21375 raft_consensus.cc:729] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8aca8b97ec4e3eb613e09d8ef6240d, State: Initialized, Role: FOLLOWER
I20260620 18:35:11.724376 21375 consensus_queue.cc:260] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.724606 21378 log.cc:824] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:11.724663 21089 heartbeater.cc:503] Master 127.10.109.126:33021 was elected leader, sending a full tablet report...
I20260620 18:35:11.724646 21375 ts_tablet_manager.cc:1428] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:11.725257 21378 tablet_bootstrap.cc:492] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: No bootstrap required, opened a new log
I20260620 18:35:11.725333 21378 ts_tablet_manager.cc:1397] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:35:11.726699 21378 raft_consensus.cc:348] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.726913 21378 raft_consensus.cc:374] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:11.726950 21378 raft_consensus.cc:729] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16efac7818ae4968934071e28121e808, State: Initialized, Role: FOLLOWER
I20260620 18:35:11.727072 21378 consensus_queue.cc:260] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.727310 21378 ts_tablet_manager.cc:1428] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:11.727356 21357 heartbeater.cc:503] Master 127.10.109.126:33021 was elected leader, sending a full tablet report...
W20260620 18:35:11.768220 21224 tablet.cc:2369] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:11.824820 21382 raft_consensus.cc:482] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:11.824963 21382 raft_consensus.cc:504] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.825438 21382 leader_election.cc:288] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d8aca8b97ec4e3eb613e09d8ef6240d (127.10.109.65:41363), 61333d01e477499fb252b0c118cd751a (127.10.109.66:33871)
I20260620 18:35:11.828544 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.825798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:60755 (local address 127.10.109.66:33871)
0620 18:35:11.825850 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.825852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.825865 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:35:11.825899 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.825900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.825911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.825970 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:11.826175 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.826900 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.827810 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.828192 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.828277 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.828286 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.828436 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.828473 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:35:11.828568 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.825665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:33871 (local address 127.10.109.67:60755)
0620 18:35:11.825731 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.825745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.825809 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.825990 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.825991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.825993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:11.826071 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.826073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.826980 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.826981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.827720 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.827722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.828275 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.828276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.828354 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.828357 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.828416 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.828460 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260620 18:35:11.828572 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.825800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:33447 (local address 127.10.109.65:41363)
0620 18:35:11.825845 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.825847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.825967 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:35:11.826003 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.826004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.826013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.826074 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:11.826265 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.826905 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.827853 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.828240 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.828321 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.828329 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.828470 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.828504 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:35:11.828590 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.825697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41363 (local address 127.10.109.67:33447)
0620 18:35:11.825888 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.825897 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:11.825960 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:11.826089 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:11.826089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:11.826091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:11.826175 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.826177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.827002 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.827003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.827760 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:11.827762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.828321 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.828322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:11.828395 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.828398 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:11.828463 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:35:11.828501 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260620 18:35:11.828925 21043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cae3f621c34134bcc38e4e7abb2d1d" candidate_uuid: "16efac7818ae4968934071e28121e808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" is_pre_election: true
I20260620 18:35:11.829005 21177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cae3f621c34134bcc38e4e7abb2d1d" candidate_uuid: "16efac7818ae4968934071e28121e808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61333d01e477499fb252b0c118cd751a" is_pre_election: true
I20260620 18:35:11.829082 21043 raft_consensus.cc:2454] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16efac7818ae4968934071e28121e808 in term 0.
I20260620 18:35:11.829187 21177 raft_consensus.cc:2454] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16efac7818ae4968934071e28121e808 in term 0.
I20260620 18:35:11.829330 21246 leader_election.cc:302] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8aca8b97ec4e3eb613e09d8ef6240d, 16efac7818ae4968934071e28121e808; no voters:
I20260620 18:35:11.829454 21382 raft_consensus.cc:2781] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:11.829509 21382 raft_consensus.cc:482] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:11.829530 21382 raft_consensus.cc:3037] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:11.830305 21382 raft_consensus.cc:504] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.830513 21382 leader_election.cc:288] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [CANDIDATE]: Term 1 election: Requested vote from peers 0d8aca8b97ec4e3eb613e09d8ef6240d (127.10.109.65:41363), 61333d01e477499fb252b0c118cd751a (127.10.109.66:33871)
I20260620 18:35:11.830596 21043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cae3f621c34134bcc38e4e7abb2d1d" candidate_uuid: "16efac7818ae4968934071e28121e808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d"
I20260620 18:35:11.830675 21043 raft_consensus.cc:3037] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:11.830698 21177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cae3f621c34134bcc38e4e7abb2d1d" candidate_uuid: "16efac7818ae4968934071e28121e808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61333d01e477499fb252b0c118cd751a"
I20260620 18:35:11.830803 21177 raft_consensus.cc:3037] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:11.831414 21043 raft_consensus.cc:2454] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16efac7818ae4968934071e28121e808 in term 1.
I20260620 18:35:11.831553 21177 raft_consensus.cc:2454] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16efac7818ae4968934071e28121e808 in term 1.
I20260620 18:35:11.831568 21246 leader_election.cc:302] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8aca8b97ec4e3eb613e09d8ef6240d, 16efac7818ae4968934071e28121e808; no voters:
I20260620 18:35:11.831700 21382 raft_consensus.cc:2781] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:11.831872 21382 raft_consensus.cc:686] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 1 LEADER]: Becoming Leader. State: Replica: 16efac7818ae4968934071e28121e808, State: Running, Role: LEADER
I20260620 18:35:11.831981 21382 consensus_queue.cc:237] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } }
I20260620 18:35:11.832787 20898 catalog_manager.cc:5495] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16efac7818ae4968934071e28121e808 (127.10.109.67). New cstate: current_term: 1 leader_uuid: "16efac7818ae4968934071e28121e808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16efac7818ae4968934071e28121e808" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 41843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:11.859774 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.857343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.10.109.67:41843)
0620 18:35:11.857425 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:11.857427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:11.857434 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:35:11.857466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:11.857467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:11.857489 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:11.857588 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:11.857742 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.858214 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.858976 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:11.859190 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:11.859265 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:11.859422 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:11.859530 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:11.859627 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:11.859648 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 18:35:11.859667 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:35:11.861617 21043 raft_consensus.cc:1261] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 1 FOLLOWER]: Refusing update from remote peer 16efac7818ae4968934071e28121e808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:11.861617 21177 raft_consensus.cc:1261] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 1 FOLLOWER]: Refusing update from remote peer 16efac7818ae4968934071e28121e808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:11.861960 21384 consensus_queue.cc:1035] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61333d01e477499fb252b0c118cd751a" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:11.862075 21382 consensus_queue.cc:1035] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d8aca8b97ec4e3eb613e09d8ef6240d" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:35:11.863566 21090 tablet.cc:2369] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:11.864462 21387 mvcc.cc:205] Tried to move back new op lower bound from 7298992176582533120 to 7298992176464404480. Current Snapshot: MvccSnapshot[applied={T|T < 7298992176582533120}]
I20260620 18:35:11.864936 21388 mvcc.cc:205] Tried to move back new op lower bound from 7298992176582533120 to 7298992176464404480. Current Snapshot: MvccSnapshot[applied={T|T < 7298992176582533120}]
I20260620 18:35:11.865123 21389 mvcc.cc:205] Tried to move back new op lower bound from 7298992176582533120 to 7298992176464404480. Current Snapshot: MvccSnapshot[applied={T|T < 7298992176582533120 or (T in {7298992176582533120})}]
I20260620 18:35:11.866871 20897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:11.866964 20897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:11.867697 20897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7049b79c3eae4c5dbd80653fd962fe16: Sending DeleteTablet for 3 replicas of tablet b7cae3f621c34134bcc38e4e7abb2d1d
I20260620 18:35:11.867990 21291 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:11 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:52540
I20260620 18:35:11.868016 21021 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:11 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:41660
I20260620 18:35:11.868320 21157 tablet_service.cc:1514] Processing DeleteTablet for tablet b7cae3f621c34134bcc38e4e7abb2d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:11 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:47592
I20260620 18:35:11.868517 21400 tablet_replica.cc:330] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: stopping tablet replica
I20260620 18:35:11.868535 21398 tablet_replica.cc:330] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d: stopping tablet replica
I20260620 18:35:11.868585 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20960
I20260620 18:35:11.868623 21398 raft_consensus.cc:2229] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:11.868724 21398 raft_consensus.cc:2258] T b7cae3f621c34134bcc38e4e7abb2d1d P 0d8aca8b97ec4e3eb613e09d8ef6240d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:11.868863 21399 tablet_replica.cc:330] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: stopping tablet replica
I20260620 18:35:11.868933 21399 raft_consensus.cc:2229] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:11.869004 21399 raft_consensus.cc:2258] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:11.868623 21400 raft_consensus.cc:2229] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:11.869140 21400 raft_consensus.cc:2258] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:11.869575 21400 ts_tablet_manager.cc:1905] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:11.869572 21399 ts_tablet_manager.cc:1905] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:11.871131 21400 ts_tablet_manager.cc:1918] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:11.871143 21399 ts_tablet_manager.cc:1918] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:11.871208 21400 log.cc:1196] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/b7cae3f621c34134bcc38e4e7abb2d1d
I20260620 18:35:11.871208 21399 log.cc:1196] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/b7cae3f621c34134bcc38e4e7abb2d1d
I20260620 18:35:11.871481 21399 ts_tablet_manager.cc:1939] T b7cae3f621c34134bcc38e4e7abb2d1d P 16efac7818ae4968934071e28121e808: Deleting consensus metadata
I20260620 18:35:11.871654 21400 ts_tablet_manager.cc:1939] T b7cae3f621c34134bcc38e4e7abb2d1d P 61333d01e477499fb252b0c118cd751a: Deleting consensus metadata
I20260620 18:35:11.871970 20883 catalog_manager.cc:4841] TS 16efac7818ae4968934071e28121e808 (127.10.109.67:41843): tablet b7cae3f621c34134bcc38e4e7abb2d1d (table test-table [id=b1d2049b42b54e0d9597ac4b1bd3cd05]) successfully deleted
I20260620 18:35:11.872104 20882 catalog_manager.cc:4841] TS 61333d01e477499fb252b0c118cd751a (127.10.109.66:33871): tablet b7cae3f621c34134bcc38e4e7abb2d1d (table test-table [id=b1d2049b42b54e0d9597ac4b1bd3cd05]) successfully deleted
W20260620 18:35:11.876034 20884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:41363 (error 108)
I20260620 18:35:11.876175 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 21094
I20260620 18:35:11.882014 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:11.881873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41363 (local address 127.0.0.1:41676)
0620 18:35:11.881930 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:11.881969 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:35:11.882087 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 21228
W20260620 18:35:11.882164 20884 catalog_manager.cc:4584] TS 0d8aca8b97ec4e3eb613e09d8ef6240d (127.10.109.65:41363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7cae3f621c34134bcc38e4e7abb2d1d: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41363: connect: Connection refused (error 111)
I20260620 18:35:11.887127 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 20866
2026-06-20T18:35:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2025 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[21404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[21404](info): set up 2 sockets
Jun 20 18:35:11 dist-test-slave-3jql krb5kdc[21404](info): commencing operation
krb5kdc: starting...
W20260620 18:35:12.939735 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:35:13 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangeradmin_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangeradmin_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangerlookup_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangerlookup_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
I20260620 18:35:13.097775 10677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx7fjtl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/postgres -l logfile start
2026-06-20 18:35:15.679 UTC [21434] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:35:15.679 UTC [21434] LOG: listening on IPv4 address "127.10.109.84", port 55245
2026-06-20 18:35:15.683 UTC [21434] LOG: listening on Unix socket "/tmp/.s.PGSQL.55245"
2026-06-20 18:35:15.717 UTC [21437] LOG: database system was shut down at 2026-06-20 18:35:14 UTC
2026-06-20 18:35:15.721 UTC [21434] LOG: database system is ready to accept connections
I20260620 18:35:16.691761 10677 mini_postgres.cc:96] Postgres bound to 55245
2026-06-20 18:35:16.697 UTC [21445] FATAL: database "slave" does not exist
127.10.109.84:55245 - accepting connections
I20260620 18:35:16.698304 10677 mini_ranger.cc:161] Starting Ranger...
I20260620 18:35:16.713248 10677 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:35:17.211509 10677 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:35:17.211596 10677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin
2026-06-20 18:35:17,505 [I] DB FLAVOR :POSTGRES
2026-06-20 18:35:17,505 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:35:17,505 [I] Checking connection..
2026-06-20 18:35:17,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:35:17,751 [I] Checking connection passed.
2026-06-20 18:35:17,751 [I] --------- Verifying version history table ---------
2026-06-20 18:35:17,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:35:18,008 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:35:18,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:35:18,240 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:35:18,240 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:35:18,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:35:18.486 UTC [21509] WARNING: there is no transaction in progress
2026-06-20 18:35:18,492 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:35:18,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:35:18,726 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:35:18,726 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:35:18,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:35:18,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:35:19,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3jql.c.gcp-upstream.internal','N') ;"
2026-06-20 18:35:19,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:35:19,675 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:35:19,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:35:19,921 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:35:19,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:35:20,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:35:20,381 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:35:20,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:35:21.321 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.337 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.354 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.370 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.388 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.408 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.473 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.480 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.495 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.505 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.516 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.530 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.539 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.545 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.553 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:21.560 UTC [21617] WARNING: there is no transaction in progress
2026-06-20 18:35:22,260 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:35:22,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:35:22,486 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:35:22,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:35:22,719 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:35:22,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:35:22,980 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:35:22,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:35:23,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:35:23,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3jql.c.gcp-upstream.internal';"
2026-06-20 18:35:23,710 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:35:23,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:35:23,935 [I] DB_PATCHES have already been applied
I20260620 18:35:23.941113 10677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx7fjtl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:35:23.941210 10677 mini_ranger.cc:193] Using host: 127.10.109.84
I20260620 18:35:23.944185 10677 mini_ranger.cc:227] Ranger admin URL: http://127.10.109.84:57697
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:35:25.057099 10677 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.001s sys 0.007s
Jun 20, 2026 6:35:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:35:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:35:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:35:45.694193 10677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangerkms_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/rangerkms_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.109.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.109.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:35:45 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:35:45.779481 10677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:35:45.788712 10677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:35:46.727257 10677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:35:46.727345 10677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms
I20260620 18:35:46.727366 10677 mini_ranger_kms.cc:227] Using postgres at 127.10.109.84:55245
2026-06-20 18:35:46,892 [I] DB FLAVOR :POSTGRES
2026-06-20 18:35:46,893 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:35:46,893 [I] Checking connection
2026-06-20 18:35:46,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:35:47,151 [I] connection success
2026-06-20 18:35:47,151 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:35:47,151 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:35:47,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:35:47,391 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:35:47,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:35:47,391 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:35:47,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:55245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:35:47,637 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:35:48.064805 10677 mini_ranger_kms.cc:310] Created kms service
I20260620 18:35:48.253083 10677 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:35:48.293885 10677 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:35:48 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:35:48.676638 10677 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:35:48.676754 10677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:35:48.676776 10677 mini_ranger_kms.cc:240] Using host: 127.10.109.84
I20260620 18:35:48.680419 10677 mini_ranger_kms.cc:276] Ranger KMS PID: 21797
I20260620 18:35:48.680539 10677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.109.84:35139
Jun 20, 2026 6:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:35:50 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:50.985194 10677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.206s user 0.005s sys 0.144s
I20260620 18:35:50.985316 10677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:35:50.985409 10677 mini_ranger_kms.cc:401] 127.10.109.84:35139/kms/v1/keys
Jun 20 18:35:51 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
Jun 20 18:35:51 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
Jun 20 18:35:53 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:35:53.950031 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41267
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:41267
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:35139/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:54.057266 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:54.057512 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:54.057550 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:54.058732 21848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:54.058833 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:54.058857 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:54.058878 21848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:54.058907 21848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:54.060436 21848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.109.84:35139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:41267
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:54.060703 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:54.060926 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:54.067133 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:54.067205 21848 server_base.cc:1034] running on GCE node
W20260620 18:35:54.067437 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:54.067487 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:54.067926 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:54.068244 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:54.069442 21848 hybrid_clock.cc:648] HybridClock initialized: now 1781980554069415 us; error 44 us; skew 500 ppm
Jun 20 18:35:54 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980554, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:54.075724 21848 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:35:54.076241 21848 webserver.cc:460] Webserver started at http://127.10.109.126:37125/ using document root <none> and password file <none>
I20260620 18:35:54.076475 21848 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:54.076536 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:54.076699 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:54 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980554, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:35:54.126986 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "1cb49f4d62d54a119f765aad72138b1a"
format_stamp: "Formatted at 2026-06-20 18:35:54 on dist-test-slave-3jql"
server_key: "94028bc3e1b5368626b7569f2b3bda8c"
server_key_iv: "f29cff4eaa6c1c33429422d66950a867"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:54.127389 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "1cb49f4d62d54a119f765aad72138b1a"
format_stamp: "Formatted at 2026-06-20 18:35:54 on dist-test-slave-3jql"
server_key: "94028bc3e1b5368626b7569f2b3bda8c"
server_key_iv: "f29cff4eaa6c1c33429422d66950a867"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:54.128834 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:35:54.143358 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:54.143627 21848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:35:54.143745 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "1cb49f4d62d54a119f765aad72138b1a"
format_stamp: "Formatted at 2026-06-20 18:35:54 on dist-test-slave-3jql"
server_key: "94028bc3e1b5368626b7569f2b3bda8c"
server_key_iv: "f29cff4eaa6c1c33429422d66950a867"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:54.143858 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:54.174156 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:54.176628 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:54.176821 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:54.181944 21848 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:41267
I20260620 18:35:54.181967 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:41267 every 8 connection(s)
I20260620 18:35:54.182335 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:54.183086 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:54.185380 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: Bootstrap starting.
I20260620 18:35:54.186718 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:54.187065 21916 log.cc:824] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:54.187093 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 21848
I20260620 18:35:54.187240 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:35:54.187458 10677 external_mini_cluster.cc:1428] Setting key be28a1e9cb9f1cac0c9d7cb50111f0a6
I20260620 18:35:54.187824 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: No bootstrap required, opened a new log
I20260620 18:35:54.190160 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } }
I20260620 18:35:54.190338 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:54.190400 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb49f4d62d54a119f765aad72138b1a, State: Initialized, Role: FOLLOWER
I20260620 18:35:54.190562 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } }
I20260620 18:35:54.190667 21916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:54.190702 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:54.190816 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:54.191573 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } }
I20260620 18:35:54.191695 21916 leader_election.cc:302] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb49f4d62d54a119f765aad72138b1a; no voters:
I20260620 18:35:54.191905 21916 leader_election.cc:288] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:54.191941 21921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:54.192180 21921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [term 1 LEADER]: Becoming Leader. State: Replica: 1cb49f4d62d54a119f765aad72138b1a, State: Running, Role: LEADER
I20260620 18:35:54.192281 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:54.192312 21921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } }
I20260620 18:35:54.192685 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cb49f4d62d54a119f765aad72138b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } } }
I20260620 18:35:54.192802 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:54.192961 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cb49f4d62d54a119f765aad72138b1a. Latest consensus state: current_term: 1 leader_uuid: "1cb49f4d62d54a119f765aad72138b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb49f4d62d54a119f765aad72138b1a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41267 } } }
I20260620 18:35:54.193061 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:54.194307 21916 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:35:54.194356 21925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:54.194869 21925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:35:54 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:54.201840 21925 catalog_manager.cc:1347] Generated new cluster ID: ebeb125bd8ed41d8aef3467883493759
I20260620 18:35:54.201946 21925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:54.203646 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:54.188147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.10.109.126:41267)
0620 18:35:54.188428 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:54.188441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:54.188487 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:35:54.190190 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:54.190203 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:54.190295 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:54.190507 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:54.191401 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:54.193064 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:54.194313 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:54.194971 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:54.195086 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:54.198114 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:54.198160 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:54.198161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:54.198263 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:54.201167 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:54.201565 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:54.201568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:54.201568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:54.201720 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:54.201916 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:54.201917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:54.201917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:54.202372 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:54.202528 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:54.203056 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 18:35:54.203356 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
I20260620 18:35:54.213524 21925 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:54.214322 21925 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:54.219188 21925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: Generated new TSK 0
I20260620 18:35:54.219421 21925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:35:54.963779 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:35139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:55.070713 21961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:55.071031 21961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:55.071074 21961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:55.072332 21961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:55.072412 21961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:55.072455 21961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:35:55.073956 21961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.109.84:35139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:55.074215 21961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:55.074455 21961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:55.080564 21967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.080646 21966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.080572 21969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:55.081236 21961 server_base.cc:1034] running on GCE node
I20260620 18:35:55.081457 21961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:55.081725 21961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:55.082878 21961 hybrid_clock.cc:648] HybridClock initialized: now 1781980555082858 us; error 50 us; skew 500 ppm
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:55.085315 21961 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:35:55.085862 21961 webserver.cc:460] Webserver started at http://127.10.109.65:46023/ using document root <none> and password file <none>
I20260620 18:35:55.086107 21961 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:55.086169 21961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:55.086356 21961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:35:55.099992 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "8683143f508e4ffaac4dd3da52529ae9"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "577f3e161aa0c878c0465c3115e630e2"
server_key_iv: "f2613c7bd002a295d67b5ccbb523147b"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.100541 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "8683143f508e4ffaac4dd3da52529ae9"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "577f3e161aa0c878c0465c3115e630e2"
server_key_iv: "f2613c7bd002a295d67b5ccbb523147b"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.102309 21961 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:35:55.122890 21978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.123111 21961 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 18:35:55.123196 21961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "8683143f508e4ffaac4dd3da52529ae9"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "577f3e161aa0c878c0465c3115e630e2"
server_key_iv: "f2613c7bd002a295d67b5ccbb523147b"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.123286 21961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:55.145874 21961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:55.148751 21961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:55.148964 21961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:55.149230 21961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:55.149647 21961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:55.149703 21961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.149791 21961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:55.149880 21961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.157243 21961 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:41831
I20260620 18:35:55.157673 21961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:55.158999 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:41831 every 8 connection(s)
I20260620 18:35:55.160333 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 21961
I20260620 18:35:55.160447 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:55.160638 10677 external_mini_cluster.cc:1428] Setting key 7d55143c308ae252ea6c761b3fcc1ac8
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:55.171048 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.158436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:60099 (local address 127.10.109.126:41267)
0620 18:35:55.160089 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.160091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.160099 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:35:55.160149 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.160150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.160164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.160226 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.160971 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.161749 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.163252 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.163571 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.163631 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.166255 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.166257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.166257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:55.166264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.168090 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.168590 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.168591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.168591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.168650 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.168925 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.168926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.168926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.169050 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.169091 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.170861 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.170948 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1624,"thread_start_us":76,"threads_started":1}
I20260620 18:35:55.172529 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.158397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41267 (local address 127.10.109.65:60099)
0620 18:35:55.159256 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.159302 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.159909 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.160351 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.160355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.160576 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:55.160857 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.160864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.161856 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.161857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.163152 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.163155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.163694 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.163695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.163829 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.164391 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:55.164402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:55.166138 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:55.168184 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.168190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.168192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.168509 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.168714 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.168715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.168715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.168831 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.169136 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:55.169142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:55.169291 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.172106 (+ 2815us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.172270 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":78,"threads_started":1}
I20260620 18:35:55.173264 22092 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41267
I20260620 18:35:55.173430 22092 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:55.173671 22092 heartbeater.cc:511] Master 127.10.109.126:41267 requested a full tablet report, sending...
I20260620 18:35:55.174242 21880 ts_manager.cc:194] Registered new tserver with Master: 8683143f508e4ffaac4dd3da52529ae9 (127.10.109.65:41831)
I20260620 18:35:55.175231 21880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:60099
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:35:55.242244 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:35139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:35:55.370066 22100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:55.370270 22100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:55.370293 22100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:55.371513 22100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:55.371553 22100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:55.371579 22100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:35:55.373041 22100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.109.84:35139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:55.373275 22100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:55.373497 22100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:55.379251 22108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.379416 22106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.379288 22105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:55.379997 22100 server_base.cc:1034] running on GCE node
I20260620 18:35:55.380220 22100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:55.380476 22100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:55.381706 22100 hybrid_clock.cc:648] HybridClock initialized: now 1781980555381683 us; error 36 us; skew 500 ppm
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:55.384928 22100 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:35:55.385478 22100 webserver.cc:460] Webserver started at http://127.10.109.66:42641/ using document root <none> and password file <none>
I20260620 18:35:55.385773 22100 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:55.385913 22100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:55.386094 22100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:35:55.399794 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "82b416632a304c06b5e0c1bebd400cfd"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "a9dd15846c132c5c0cb3fccdc2e91c02"
server_key_iv: "d67c4050fe54e2ee260be08b3bf69b77"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.400327 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "82b416632a304c06b5e0c1bebd400cfd"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "a9dd15846c132c5c0cb3fccdc2e91c02"
server_key_iv: "d67c4050fe54e2ee260be08b3bf69b77"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.405164 22100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260620 18:35:55.422258 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.423411 22100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:35:55.423575 22100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "82b416632a304c06b5e0c1bebd400cfd"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "a9dd15846c132c5c0cb3fccdc2e91c02"
server_key_iv: "d67c4050fe54e2ee260be08b3bf69b77"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.423755 22100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:55.440353 22100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:55.442672 22100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:55.442912 22100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:55.443215 22100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:55.443595 22100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:55.443682 22100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.443759 22100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:55.443822 22100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.450325 22100 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:35693
I20260620 18:35:55.450790 22100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:55.453449 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:35693 every 8 connection(s)
I20260620 18:35:55.460547 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 22100
I20260620 18:35:55.460657 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:55.460846 10677 external_mini_cluster.cc:1428] Setting key 83f73fae463906762699d6e7e8c33628
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:55.465541 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.454295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:49039 (local address 127.10.109.126:41267)
0620 18:35:55.454354 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.454356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.454788 (+ 432us) server_negotiation.cc:407] Connection header received
0620 18:35:55.455639 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.455640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.455655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.455687 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.456496 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.457258 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.458759 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.459065 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.459112 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.462231 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.462233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.462233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:55.462240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.463946 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.464447 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.464448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.464448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.464506 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.464797 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.464798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.464798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.464930 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.464972 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.465353 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.465441 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:35:55.466158 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.453996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41267 (local address 127.10.109.66:49039)
0620 18:35:55.454611 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.454656 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.455515 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.455863 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.455867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.456095 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:55.456380 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.456386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.457377 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.457378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.458643 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.458646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.459221 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.459222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.459358 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.459900 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:55.459910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:55.462115 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:55.464046 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.464052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.464054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.464348 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.464583 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.464584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.464584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.464710 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.465020 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:55.465026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:55.465175 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.465766 (+ 591us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.465943 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":95,"threads_started":1}
I20260620 18:35:55.466694 22230 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41267
I20260620 18:35:55.466804 22230 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:55.466980 22230 heartbeater.cc:511] Master 127.10.109.126:41267 requested a full tablet report, sending...
I20260620 18:35:55.467365 21880 ts_manager.cc:194] Registered new tserver with Master: 82b416632a304c06b5e0c1bebd400cfd (127.10.109.66:35693)
I20260620 18:35:55.467820 21880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:49039
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:35:55.501731 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:35139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:55.620577 22235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:55.620900 22235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:55.620983 22235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:55.622660 22235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:55.622797 22235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:55.622887 22235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:35:55.625036 22235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.109.84:35139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:35:55.625463 22235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:55.625770 22235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:55.632836 22241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.632891 22240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.633468 22243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:55.634198 22235 server_base.cc:1034] running on GCE node
I20260620 18:35:55.634405 22235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:55.634668 22235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:55.642041 22235 hybrid_clock.cc:648] HybridClock initialized: now 1781980555642003 us; error 49 us; skew 500 ppm
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:55.644833 22235 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:35:55.645260 22235 webserver.cc:460] Webserver started at http://127.10.109.67:44197/ using document root <none> and password file <none>
I20260620 18:35:55.645557 22235 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:55.645630 22235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:55.645776 22235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:35:55.658404 22235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "f521e6a5fc364e7ba45a7896e92624ff"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "f71e8f8c75c24b4ac9bcf5fb2ae0732c"
server_key_iv: "aa7e585b66b2a9dba0dbda259ea92bbb"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.658972 22235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "f521e6a5fc364e7ba45a7896e92624ff"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "f71e8f8c75c24b4ac9bcf5fb2ae0732c"
server_key_iv: "aa7e585b66b2a9dba0dbda259ea92bbb"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.663156 22235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260620 18:35:55.682775 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.682999 22235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 18:35:55.683094 22235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "f521e6a5fc364e7ba45a7896e92624ff"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-3jql"
server_key: "f71e8f8c75c24b4ac9bcf5fb2ae0732c"
server_key_iv: "aa7e585b66b2a9dba0dbda259ea92bbb"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:55.683190 22235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:55.704232 22235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:55.706777 22235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:55.706971 22235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:55.707279 22235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:55.707679 22235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:55.707770 22235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.707847 22235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:55.707908 22235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.714900 22235 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40199
I20260620 18:35:55.715378 22235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:55.719599 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 22235
I20260620 18:35:55.719708 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:55.719933 10677 external_mini_cluster.cc:1428] Setting key dd34a5a65fe86160e396dfd100ca5906
I20260620 18:35:55.722044 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40199 every 8 connection(s)
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:35:55.736331 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.722759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:54191 (local address 127.10.109.126:41267)
0620 18:35:55.722801 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.722803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.723076 (+ 273us) server_negotiation.cc:407] Connection header received
0620 18:35:55.723760 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.723761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.723773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.723806 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.724667 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.725439 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.726947 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.727254 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.727299 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.730126 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.730127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.730128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:55.730135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.732477 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.735254 (+ 2777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.735255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.735256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.735314 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.735601 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.735602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.735602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.735728 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.735792 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.736150 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.736242 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:35:55.737052 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.722726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41267 (local address 127.10.109.67:54191)
0620 18:35:55.722945 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.722995 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.723633 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.723990 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.723994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.724289 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:55.724556 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.724562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.725556 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.725556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.726848 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.726851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.727387 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.727388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.727522 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.728189 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:55.728210 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:55.730017 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:55.734831 (+ 4814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.734837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.734840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.735139 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.735392 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:55.735393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:55.735393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:55.735521 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:55.735826 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:55.735833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:55.735984 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.736659 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.736817 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260620 18:35:55.737721 22364 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41267
I20260620 18:35:55.737811 22364 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:55.737991 22364 heartbeater.cc:511] Master 127.10.109.126:41267 requested a full tablet report, sending...
I20260620 18:35:55.738386 21880 ts_manager.cc:194] Registered new tserver with Master: f521e6a5fc364e7ba45a7896e92624ff (127.10.109.67:40199)
I20260620 18:35:55.738862 21880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:54191
I20260620 18:35:55.742254 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:55.756045 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.743931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.10.109.126:41267)
0620 18:35:55.743968 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.743969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.744815 (+ 846us) server_negotiation.cc:407] Connection header received
0620 18:35:55.744916 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.744917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.744927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.744956 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.748139 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.748872 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.750064 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.750344 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.750389 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.752061 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.752063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.752063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:55.752069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.753728 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.755212 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.755213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.755214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.755273 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.755487 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.755488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.755488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.755602 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.755683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.755882 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.755956 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:35:55.758234 21880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:55.760500 21880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:55.778347 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.764680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:40199 (local address 127.0.0.1:55120)
0620 18:35:55.766605 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.766645 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.766725 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.767139 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.767142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.767145 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:55.767205 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.767207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.775070 (+ 7863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.775071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.776143 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.776146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.778058 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.778060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.778140 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.778153 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.778216 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.778268 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":84,"threads_started":1}
I20260620 18:35:55.779040 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.765199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.10.109.67:40199)
0620 18:35:55.766912 (+ 1713us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.766917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.766926 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:55.766962 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.766964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.766980 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.767052 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:55.768465 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.774969 (+ 6504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.776290 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.777065 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.778812 (+ 1747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.778896 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.778922 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.778963 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1675,"thread_start_us":74,"threads_started":1}
I20260620 18:35:55.779491 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.764895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41831 (local address 127.0.0.1:53948)
0620 18:35:55.770450 (+ 5555us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.770460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.770504 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.770778 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.770779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.770781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:55.770826 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.770828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.773561 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.773562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.774878 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.774881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.779263 (+ 4382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.779264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.779334 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.779337 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.779382 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.779420 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5539,"thread_start_us":70,"threads_started":1}
I20260620 18:35:55.780346 22298 tablet_service.cc:1467] Processing CreateTablet for tablet 899c4a2c80de4279918d678a066e3412 (DEFAULT_TABLE table=test-table [id=a6a4836f1035426caa3666e7f6f49559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:55.780717 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c4a2c80de4279918d678a066e3412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:55.781420 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.765810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.10.109.66:35693)
0620 18:35:55.771026 (+ 5216us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.771031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.771040 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:55.771077 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.771079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.771094 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.771158 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:55.771396 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.778780 (+ 7384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.780113 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.780853 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.780920 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.780982 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.781306 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.781345 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5180,"thread_start_us":84,"threads_started":1}
I20260620 18:35:55.781229 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.765579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35693 (local address 127.0.0.1:35530)
0620 18:35:55.770831 (+ 5252us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.770843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.770902 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.771229 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.771229 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.771231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:55.771320 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.771321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.778877 (+ 7556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.778878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.779999 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.780001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.781016 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.781017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.781078 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.781081 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.781127 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.781163 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5234,"thread_start_us":60,"threads_started":1}
I20260620 18:35:55.784900 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.765351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.10.109.65:41831)
0620 18:35:55.769682 (+ 4331us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.769686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.770560 (+ 874us) server_negotiation.cc:407] Connection header received
0620 18:35:55.770598 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.770600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.770616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.770679 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:55.772430 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.773463 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.777151 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.777870 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.777943 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.778002 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.784773 (+ 6771us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.784822 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4295,"thread_start_us":47,"threads_started":1}
I20260620 18:35:55.785511 22164 tablet_service.cc:1467] Processing CreateTablet for tablet 899c4a2c80de4279918d678a066e3412 (DEFAULT_TABLE table=test-table [id=a6a4836f1035426caa3666e7f6f49559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:55.785833 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c4a2c80de4279918d678a066e3412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:55.786880 22383 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Bootstrap starting.
I20260620 18:35:55.789003 22383 tablet_bootstrap.cc:654] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:55.789283 22383 log.cc:824] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:55.789736 22384 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Bootstrap starting.
I20260620 18:35:55.789994 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 899c4a2c80de4279918d678a066e3412 (DEFAULT_TABLE table=test-table [id=a6a4836f1035426caa3666e7f6f49559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:55.790311 22008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899c4a2c80de4279918d678a066e3412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:55.790704 22384 tablet_bootstrap.cc:654] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:55.791018 22384 log.cc:824] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:55.793404 22387 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Bootstrap starting.
I20260620 18:35:55.794109 22387 tablet_bootstrap.cc:654] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:55.794193 22383 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: No bootstrap required, opened a new log
I20260620 18:35:55.794306 22383 ts_tablet_manager.cc:1397] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 18:35:55.794471 22387 log.cc:824] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:55.795300 22384 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: No bootstrap required, opened a new log
I20260620 18:35:55.795404 22384 ts_tablet_manager.cc:1397] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 18:35:55.796538 22383 raft_consensus.cc:348] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.796926 22383 raft_consensus.cc:374] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:55.797017 22383 raft_consensus.cc:729] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f521e6a5fc364e7ba45a7896e92624ff, State: Initialized, Role: FOLLOWER
I20260620 18:35:55.797205 22383 consensus_queue.cc:260] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.797344 22384 raft_consensus.cc:348] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.797482 22384 raft_consensus.cc:374] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:55.797508 22384 raft_consensus.cc:729] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b416632a304c06b5e0c1bebd400cfd, State: Initialized, Role: FOLLOWER
I20260620 18:35:55.797626 22384 consensus_queue.cc:260] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.797830 22383 ts_tablet_manager.cc:1428] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:55.797843 22384 ts_tablet_manager.cc:1428] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:35:55.797946 22230 heartbeater.cc:503] Master 127.10.109.126:41267 was elected leader, sending a full tablet report...
I20260620 18:35:55.797974 22364 heartbeater.cc:503] Master 127.10.109.126:41267 was elected leader, sending a full tablet report...
I20260620 18:35:55.800282 22387 tablet_bootstrap.cc:492] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: No bootstrap required, opened a new log
I20260620 18:35:55.800400 22387 ts_tablet_manager.cc:1397] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 18:35:55.802819 22387 raft_consensus.cc:348] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.802965 22387 raft_consensus.cc:374] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:55.803035 22387 raft_consensus.cc:729] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8683143f508e4ffaac4dd3da52529ae9, State: Initialized, Role: FOLLOWER
I20260620 18:35:55.803210 22387 consensus_queue.cc:260] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.803507 22387 ts_tablet_manager.cc:1428] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:55.803655 22092 heartbeater.cc:503] Master 127.10.109.126:41267 was elected leader, sending a full tablet report...
Jun 20 18:35:55 dist-test-slave-3jql krb5kdc[21404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
W20260620 18:35:55.909713 22093 tablet.cc:2369] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:55.954821 22231 tablet.cc:2369] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:55.968804 22389 raft_consensus.cc:482] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:55.969089 22389 raft_consensus.cc:504] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.969539 22389 leader_election.cc:288] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8683143f508e4ffaac4dd3da52529ae9 (127.10.109.65:41831), 82b416632a304c06b5e0c1bebd400cfd (127.10.109.66:35693)
W20260620 18:35:55.972702 22365 tablet.cc:2369] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:55.973618 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.969695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41831 (local address 127.10.109.67:39885)
0620 18:35:55.969798 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.969811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.969857 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.970189 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.970189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.970191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:55.970260 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.970262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.971240 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.971242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.972497 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.972500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.973363 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.973363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.973442 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.973445 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.973492 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.973536 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":74}
I20260620 18:35:55.973618 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.969886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:50361 (local address 127.10.109.66:35693)
0620 18:35:55.969926 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.969928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.970096 (+ 168us) server_negotiation.cc:407] Connection header received
0620 18:35:55.970196 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.970197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.970208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.970240 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:55.970440 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.971157 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.972619 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.973216 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.973260 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.973269 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.973525 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.973558 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:35:55.973862 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.969714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35693 (local address 127.10.109.67:50361)
0620 18:35:55.970040 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:55.970050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:55.970122 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:55.970300 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:55.970301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:55.970302 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:55.970368 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.970369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.971248 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.971249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.972512 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:55.972514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.973370 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.973371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:55.973442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.973445 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:35:55.973741 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:35:55.973792 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20260620 18:35:55.974184 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.969938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39885 (local address 127.10.109.65:41831)
0620 18:35:55.969983 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.969985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.969994 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:55.970021 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.970023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.970033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.970086 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:55.970337 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.971129 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.972616 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.973200 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.973267 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.973276 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.974063 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.974109 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:35:55.974218 22181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c4a2c80de4279918d678a066e3412" candidate_uuid: "f521e6a5fc364e7ba45a7896e92624ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b416632a304c06b5e0c1bebd400cfd" is_pre_election: true
I20260620 18:35:55.974360 22181 raft_consensus.cc:2454] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f521e6a5fc364e7ba45a7896e92624ff in term 0.
I20260620 18:35:55.974443 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c4a2c80de4279918d678a066e3412" candidate_uuid: "f521e6a5fc364e7ba45a7896e92624ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8683143f508e4ffaac4dd3da52529ae9" is_pre_election: true
I20260620 18:35:55.974576 22039 raft_consensus.cc:2454] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f521e6a5fc364e7ba45a7896e92624ff in term 0.
I20260620 18:35:55.974594 22252 leader_election.cc:302] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b416632a304c06b5e0c1bebd400cfd, f521e6a5fc364e7ba45a7896e92624ff; no voters:
I20260620 18:35:55.974716 22389 raft_consensus.cc:2781] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:55.974786 22389 raft_consensus.cc:482] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:55.974812 22389 raft_consensus.cc:3037] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:55.975584 22389 raft_consensus.cc:504] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.975733 22389 leader_election.cc:288] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [CANDIDATE]: Term 1 election: Requested vote from peers 8683143f508e4ffaac4dd3da52529ae9 (127.10.109.65:41831), 82b416632a304c06b5e0c1bebd400cfd (127.10.109.66:35693)
I20260620 18:35:55.975898 22181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c4a2c80de4279918d678a066e3412" candidate_uuid: "f521e6a5fc364e7ba45a7896e92624ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b416632a304c06b5e0c1bebd400cfd"
I20260620 18:35:55.975965 22181 raft_consensus.cc:3037] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:55.976156 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899c4a2c80de4279918d678a066e3412" candidate_uuid: "f521e6a5fc364e7ba45a7896e92624ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8683143f508e4ffaac4dd3da52529ae9"
I20260620 18:35:55.976259 22039 raft_consensus.cc:3037] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:55.976812 22181 raft_consensus.cc:2454] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f521e6a5fc364e7ba45a7896e92624ff in term 1.
I20260620 18:35:55.976971 22252 leader_election.cc:302] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b416632a304c06b5e0c1bebd400cfd, f521e6a5fc364e7ba45a7896e92624ff; no voters:
I20260620 18:35:55.977046 22389 raft_consensus.cc:2781] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:55.977139 22039 raft_consensus.cc:2454] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f521e6a5fc364e7ba45a7896e92624ff in term 1.
I20260620 18:35:55.977186 22389 raft_consensus.cc:686] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 1 LEADER]: Becoming Leader. State: Replica: f521e6a5fc364e7ba45a7896e92624ff, State: Running, Role: LEADER
I20260620 18:35:55.977272 22389 consensus_queue.cc:237] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } }
I20260620 18:35:55.978005 21880 catalog_manager.cc:5495] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f521e6a5fc364e7ba45a7896e92624ff (127.10.109.67). New cstate: current_term: 1 leader_uuid: "f521e6a5fc364e7ba45a7896e92624ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f521e6a5fc364e7ba45a7896e92624ff" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:55.995258 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.992555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.10.109.67:40199)
0620 18:35:55.992611 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.992613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.992622 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:35:55.992702 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.992703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.992722 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.992797 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:55.992959 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.993519 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.994407 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.994630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.994716 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.994917 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:55.995023 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:55.995119 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.995138 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.995157 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:35:55.996912 22039 raft_consensus.cc:1261] T 899c4a2c80de4279918d678a066e3412 P 8683143f508e4ffaac4dd3da52529ae9 [term 1 FOLLOWER]: Refusing update from remote peer f521e6a5fc364e7ba45a7896e92624ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:55.996912 22181 raft_consensus.cc:1261] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 1 FOLLOWER]: Refusing update from remote peer f521e6a5fc364e7ba45a7896e92624ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:55.997238 22393 consensus_queue.cc:1035] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b416632a304c06b5e0c1bebd400cfd" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:55.997354 22389 consensus_queue.cc:1035] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8683143f508e4ffaac4dd3da52529ae9" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:56.002229 21879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:56.002338 21879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:56.003575 21879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cb49f4d62d54a119f765aad72138b1a: Sending DeleteTablet for 3 replicas of tablet 899c4a2c80de4279918d678a066e3412
I20260620 18:35:56.003983 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c4a2c80de4279918d678a066e3412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:55 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:53948
I20260620 18:35:56.004160 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 21961
I20260620 18:35:56.004231 22164 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c4a2c80de4279918d678a066e3412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:55 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:35530
I20260620 18:35:56.004742 22298 tablet_service.cc:1514] Processing DeleteTablet for tablet 899c4a2c80de4279918d678a066e3412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:55 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55120
W20260620 18:35:56.013355 21866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:41831 (error 108)
I20260620 18:35:56.013711 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 22100
I20260620 18:35:56.013798 22406 tablet_replica.cc:330] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd: stopping tablet replica
I20260620 18:35:56.013895 22406 raft_consensus.cc:2229] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:56.013988 22406 raft_consensus.cc:2258] T 899c4a2c80de4279918d678a066e3412 P 82b416632a304c06b5e0c1bebd400cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:56.014120 22407 tablet_replica.cc:330] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: stopping tablet replica
I20260620 18:35:56.014297 22407 raft_consensus.cc:2229] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:56.014392 22407 raft_consensus.cc:2258] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:56.015388 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:56.014649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41831 (local address 127.0.0.1:53962)
0620 18:35:56.015311 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:56.015356 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
W20260620 18:35:56.015494 21866 catalog_manager.cc:4584] TS 8683143f508e4ffaac4dd3da52529ae9 (127.10.109.65:41831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 899c4a2c80de4279918d678a066e3412: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41831: connect: Connection refused (error 111)
I20260620 18:35:56.017663 22407 ts_tablet_manager.cc:1905] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:56.019630 22407 ts_tablet_manager.cc:1918] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:56.019774 22407 log.cc:1196] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/899c4a2c80de4279918d678a066e3412
I20260620 18:35:56.020166 22407 ts_tablet_manager.cc:1939] T 899c4a2c80de4279918d678a066e3412 P f521e6a5fc364e7ba45a7896e92624ff: Deleting consensus metadata
I20260620 18:35:56.020795 21866 catalog_manager.cc:4841] TS f521e6a5fc364e7ba45a7896e92624ff (127.10.109.67:40199): tablet 899c4a2c80de4279918d678a066e3412 (table test-table [id=a6a4836f1035426caa3666e7f6f49559]) successfully deleted
W20260620 18:35:56.022320 22381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:56.021950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35693 (local address 127.0.0.1:35542)
0620 18:35:56.021999 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:56.022010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:56.022046 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:56.022274 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.66:35693: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:35:56.022639 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 22235
I20260620 18:35:56.028327 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 21848
I20260620 18:35:56.033731 10677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:35:56.377758 10677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:35:56.377909 10677 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:35:56.722106 10677 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:35:56.722 UTC [21434] LOG: received smart shutdown request
2026-06-20 18:35:56.725 UTC [21434] LOG: background worker "logical replication launcher" (PID 21443) exited with exit code 1
2026-06-20 18:35:56.726 UTC [21438] LOG: shutting down
2026-06-20 18:35:56.916 UTC [21434] LOG: database system is shut down
2026-06-20T18:35:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45142 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:57 dist-test-slave-3jql krb5kdc[22417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:57 dist-test-slave-3jql krb5kdc[22417](info): set up 2 sockets
Jun 20 18:35:57 dist-test-slave-3jql krb5kdc[22417](info): commencing operation
krb5kdc: starting...
W20260620 18:35:58.070632 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:35:58 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangeradmin_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangeradmin_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangerlookup_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangerlookup_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
I20260620 18:35:58.230703 10677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx7fjtl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/postgres -l logfile start
2026-06-20 18:36:01.080 UTC [22447] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:36:01.080 UTC [22447] LOG: listening on IPv4 address "127.10.109.84", port 45381
2026-06-20 18:36:01.084 UTC [22447] LOG: listening on Unix socket "/tmp/.s.PGSQL.45381"
2026-06-20 18:36:01.105 UTC [22450] LOG: database system was shut down at 2026-06-20 18:35:59 UTC
2026-06-20 18:36:01.109 UTC [22447] LOG: database system is ready to accept connections
I20260620 18:36:02.074803 10677 mini_postgres.cc:96] Postgres bound to 45381
2026-06-20 18:36:02.080 UTC [22458] FATAL: database "slave" does not exist
127.10.109.84:45381 - accepting connections
I20260620 18:36:02.081481 10677 mini_ranger.cc:161] Starting Ranger...
I20260620 18:36:02.095461 10677 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:36:02.644951 10677 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:36:02.645025 10677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin
2026-06-20 18:36:02,919 [I] DB FLAVOR :POSTGRES
2026-06-20 18:36:02,919 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:36:02,919 [I] Checking connection..
2026-06-20 18:36:02,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:36:03,151 [I] Checking connection passed.
2026-06-20 18:36:03,151 [I] --------- Verifying version history table ---------
2026-06-20 18:36:03,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:36:03,399 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:36:03,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:36:03,645 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:36:03,645 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:36:03,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:36:03.892 UTC [22522] WARNING: there is no transaction in progress
2026-06-20 18:36:03,898 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:36:03,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:36:04,131 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:36:04,132 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:36:04,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:36:04,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:36:04,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3jql.c.gcp-upstream.internal','N') ;"
2026-06-20 18:36:04,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:36:05,055 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:36:05,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:36:05,302 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:36:05,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:36:05,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:36:05,781 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:36:05,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:36:06.748 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.764 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.782 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.798 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.815 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.834 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.903 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.910 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.926 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.936 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.947 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.961 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.970 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.977 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.984 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:06.992 UTC [22630] WARNING: there is no transaction in progress
2026-06-20 18:36:07,691 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:36:07,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:36:07,949 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:36:07,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:36:08,183 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:36:08,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:36:08,420 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:36:08,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:36:08,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:36:08,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3jql.c.gcp-upstream.internal';"
2026-06-20 18:36:09,182 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:36:09,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:36:09,430 [I] DB_PATCHES have already been applied
I20260620 18:36:09.435981 10677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx7fjtl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:36:09.436064 10677 mini_ranger.cc:193] Using host: 127.10.109.84
I20260620 18:36:09.439308 10677 mini_ranger.cc:227] Ranger admin URL: http://127.10.109.84:41339
Jun 20, 2026 6:36:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:36:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:36:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:36:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:36:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:36:10.564935 10677 mini_ranger.cc:160] Time spent starting Ranger: real 8.483s user 0.003s sys 0.005s
Jun 20, 2026 6:36:30 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:36:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:36:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:36:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:36:31 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:36:31.903383 10677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.109.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.109.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangerkms_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.109.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/rangerkms_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.109.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.109.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.109.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/HTTP_127.10.109.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:36:31 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:36:31.985014 10677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:36:31.994952 10677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:36:32.915568 10677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:36:32.915654 10677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms
I20260620 18:36:32.915678 10677 mini_ranger_kms.cc:227] Using postgres at 127.10.109.84:45381
2026-06-20 18:36:33,072 [I] DB FLAVOR :POSTGRES
2026-06-20 18:36:33,072 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:36:33,072 [I] Checking connection
2026-06-20 18:36:33,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:36:33,301 [I] connection success
2026-06-20 18:36:33,301 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:36:33,301 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:36:33,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:36:33,547 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:36:33,547 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:36:33,547 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:36:33,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.109.84:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:36:33,822 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:36:34.291270 10677 mini_ranger_kms.cc:310] Created kms service
I20260620 18:36:34.457931 10677 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:36:34.501968 10677 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:36:34 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:34.880508 10677 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:36:34.880698 10677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx7fjtl/build/release/bin/postgresql.jar:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx7fjtl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:36:34.880717 10677 mini_ranger_kms.cc:240] Using host: 127.10.109.84
I20260620 18:36:34.884007 10677 mini_ranger_kms.cc:276] Ranger KMS PID: 22810
I20260620 18:36:34.884073 10677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.109.84:40723
Jun 20, 2026 6:36:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:36:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:36:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:36:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:36:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:36:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:36:37 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:37.183144 10677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.198s user 0.016s sys 0.124s
I20260620 18:36:37.183248 10677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:36:37.183296 10677 mini_ranger_kms.cc:401] 127.10.109.84:40723/kms/v1/keys
Jun 20 18:36:37 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
Jun 20 18:36:37 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.109.84@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
Jun 20 18:36:40 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:36:40.144987 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44155
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:44155
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:40.223646 22861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:40.223834 22861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:40.223855 22861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:40.224982 22861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:40.225023 22861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:40.225037 22861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:40.225049 22861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:40.225061 22861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:40.226564 22861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.109.84:40723/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:44155
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:40.226804 22861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:40.227038 22861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:40.231618 22866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:40.231609 22867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:40.231896 22861 server_base.cc:1034] running on GCE node
W20260620 18:36:40.231930 22869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:40.232165 22861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:40.232432 22861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:40.233599 22861 hybrid_clock.cc:648] HybridClock initialized: now 1781980600233588 us; error 32 us; skew 500 ppm
Jun 20 18:36:40 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:40.236207 22861 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:40.236654 22861 webserver.cc:460] Webserver started at http://127.10.109.126:41849/ using document root <none> and password file <none>
I20260620 18:36:40.236871 22861 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:40.236930 22861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:40.237087 22861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:36:40 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:40.296051 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "c3f64c627c8a4b7aacad45a589e56073"
format_stamp: "Formatted at 2026-06-20 18:36:40 on dist-test-slave-3jql"
server_key: "a91657072f1a6491579c72aacf213b15"
server_key_iv: "47c9d3325304a011b946a2b32d81fc2c"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:40.296535 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "c3f64c627c8a4b7aacad45a589e56073"
format_stamp: "Formatted at 2026-06-20 18:36:40 on dist-test-slave-3jql"
server_key: "a91657072f1a6491579c72aacf213b15"
server_key_iv: "47c9d3325304a011b946a2b32d81fc2c"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:40.298062 22861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:36:40.312301 22876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.312517 22861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:40.312611 22861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "c3f64c627c8a4b7aacad45a589e56073"
format_stamp: "Formatted at 2026-06-20 18:36:40 on dist-test-slave-3jql"
server_key: "a91657072f1a6491579c72aacf213b15"
server_key_iv: "47c9d3325304a011b946a2b32d81fc2c"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:40.312700 22861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:40.330120 22861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:40.331888 22861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:40.332003 22861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:40.336285 22861 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:44155
I20260620 18:36:40.336354 22928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:44155 every 8 connection(s)
I20260620 18:36:40.336684 22861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:40.337350 22929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:40.339656 22929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Bootstrap starting.
I20260620 18:36:40.340055 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 22861
I20260620 18:36:40.340142 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:36:40.340305 22929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:40.340356 10677 external_mini_cluster.cc:1428] Setting key 833c7d2d05304ebb7db65880e50b113f
I20260620 18:36:40.340595 22929 log.cc:824] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:40.341336 22929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: No bootstrap required, opened a new log
I20260620 18:36:40.343758 22929 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:40.343933 22929 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:40.344017 22929 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f64c627c8a4b7aacad45a589e56073, State: Initialized, Role: FOLLOWER
I20260620 18:36:40.344131 22929 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:40.344199 22929 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:40.344226 22929 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:40.344260 22929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:40.344887 22929 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:40.345010 22929 leader_election.cc:302] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f64c627c8a4b7aacad45a589e56073; no voters:
I20260620 18:36:40.345182 22929 leader_election.cc:288] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:40.345216 22934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:40.345366 22934 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 LEADER]: Becoming Leader. State: Replica: c3f64c627c8a4b7aacad45a589e56073, State: Running, Role: LEADER
I20260620 18:36:40.345466 22929 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:40.345500 22934 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:40.345948 22935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f64c627c8a4b7aacad45a589e56073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } } }
I20260620 18:36:40.346025 22935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:40.346194 22936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f64c627c8a4b7aacad45a589e56073. Latest consensus state: current_term: 1 leader_uuid: "c3f64c627c8a4b7aacad45a589e56073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } } }
I20260620 18:36:40.346246 22936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:40.346714 22940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:40.347147 22940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:36:40 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:40.348495 22929 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:36:40.351922 22940 catalog_manager.cc:1347] Generated new cluster ID: af32d1b3134947d29b93bb1b94fceb84
I20260620 18:36:40.352089 22940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:40.354839 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.340883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.10.109.126:44155)
0620 18:36:40.341047 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.341057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.341090 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:36:40.341554 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.341560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.341604 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.341715 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.342528 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.343433 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.344517 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.344968 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.345133 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.348025 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:40.348028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:40.348029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:40.348039 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:40.352356 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:40.352852 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:40.352856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:40.352857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:40.353011 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:40.353223 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:40.353224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:40.353224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:40.353644 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:40.353829 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:40.353896 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:36:40.354250 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 18:36:40.367220 22940 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:40.367991 22940 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:40.371950 22940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Generated new TSK 0
I20260620 18:36:40.372121 22940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:36:41.046258 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:41.131003 22974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:41.131209 22974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:41.131243 22974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:41.132357 22974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:41.132423 22974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:41.132468 22974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:36:41.133956 22974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:41.134194 22974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:41.134409 22974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:41.139377 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.139410 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:41.139595 22974 server_base.cc:1034] running on GCE node
W20260620 18:36:41.139420 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:41.139931 22974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:41.140177 22974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:41.141367 22974 hybrid_clock.cc:648] HybridClock initialized: now 1781980601141348 us; error 48 us; skew 500 ppm
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:41.144207 22974 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:36:41.144613 22974 webserver.cc:460] Webserver started at http://127.10.109.65:44125/ using document root <none> and password file <none>
I20260620 18:36:41.144862 22974 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:41.144933 22974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:41.145076 22974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:41.162617 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "815632fac60d4a7d8b00d9148443d6ee"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "8128504149beac1b5f2b9c126209e1dc"
server_key_iv: "9b03f27aa5d7d9b3d7c78b7dfa9ebd10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.163129 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "815632fac60d4a7d8b00d9148443d6ee"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "8128504149beac1b5f2b9c126209e1dc"
server_key_iv: "9b03f27aa5d7d9b3d7c78b7dfa9ebd10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.166862 22974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:41.180090 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.180279 22974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:41.180363 22974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "815632fac60d4a7d8b00d9148443d6ee"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "8128504149beac1b5f2b9c126209e1dc"
server_key_iv: "9b03f27aa5d7d9b3d7c78b7dfa9ebd10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.180437 22974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:41.192102 22974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:41.194334 22974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:41.194531 22974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:41.194784 22974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:41.195153 22974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:41.195232 22974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.195292 22974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:41.195336 22974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.202478 22974 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:44617
I20260620 18:36:41.202543 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:44617 every 8 connection(s)
I20260620 18:36:41.202926 22974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:41.204270 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 22974
I20260620 18:36:41.204373 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:41.204555 10677 external_mini_cluster.cc:1428] Setting key ab027a6b639486317501b6384823cbf6
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:41.223989 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.203631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.65:49853)
0620 18:36:41.203812 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.203857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.204450 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.214829 (+ 10379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.214833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.215003 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.215242 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.215247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.216140 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.216141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.217129 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.217132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.217674 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.217676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.217821 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.219133 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.219144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.220707 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:41.222630 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.222635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.222637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.222919 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.223131 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.223132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.223132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.223225 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.223523 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:41.223527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:41.223633 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 18:36:41.223705 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:36:41.223808 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260620 18:36:41.223984 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.203849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:49853 (local address 127.10.109.126:44155)
0620 18:36:41.214528 (+ 10679us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.214530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.214540 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:36:41.214567 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.214568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.214580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.214665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.215361 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.215951 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.217237 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.217543 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.218070 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.220797 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:41.220799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:41.220799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:41.220807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:41.222540 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.222990 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.222991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.222992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.223059 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.223296 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.223297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.223298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.223450 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:41.223529 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:41.223770 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:36:41.223859 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10647,"thread_start_us":77,"threads_started":1}
I20260620 18:36:41.224431 23103 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:41.224517 23103 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:41.224740 23103 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:41.225409 22893 ts_manager.cc:194] Registered new tserver with Master: 815632fac60d4a7d8b00d9148443d6ee (127.10.109.65:44617)
I20260620 18:36:41.226336 22893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:49853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:36:41.237969 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:41.336949 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:41.337260 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:41.337338 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:41.339082 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:41.339175 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:41.339251 23111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:36:41.341369 23111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:41.341830 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:41.342125 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:41.348444 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.348415 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.348891 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:41.349387 23111 server_base.cc:1034] running on GCE node
I20260620 18:36:41.349608 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:41.349857 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:41.351015 23111 hybrid_clock.cc:648] HybridClock initialized: now 1781980601350996 us; error 32 us; skew 500 ppm
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:41.354156 23111 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:36:41.354626 23111 webserver.cc:460] Webserver started at http://127.10.109.66:46371/ using document root <none> and password file <none>
I20260620 18:36:41.354938 23111 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:41.355005 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:41.355144 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:41.373044 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "3e8fcb9d61289569367da7d95cd4d26f"
server_key_iv: "677e244ce2ff716df26444dfdbb1a6df"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.373539 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "3e8fcb9d61289569367da7d95cd4d26f"
server_key_iv: "677e244ce2ff716df26444dfdbb1a6df"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.375590 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 18:36:41.392009 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.392239 23111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:41.392331 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "3e8fcb9d61289569367da7d95cd4d26f"
server_key_iv: "677e244ce2ff716df26444dfdbb1a6df"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.392409 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:41.446208 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:41.452411 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:41.452559 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:41.452760 23111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:41.453104 23111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:41.453150 23111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.453190 23111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:41.453210 23111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.459537 23111 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37735
I20260620 18:36:41.459990 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:41.464828 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23111
I20260620 18:36:41.464960 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:41.465217 10677 external_mini_cluster.cc:1428] Setting key 14a5e1b74b02bf431c578df376fef845
I20260620 18:36:41.468405 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37735 every 8 connection(s)
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:41.475754 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.461254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:54181 (local address 127.10.109.126:44155)
0620 18:36:41.461312 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.461325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.461610 (+ 285us) server_negotiation.cc:407] Connection header received
0620 18:36:41.462984 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.462986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.463002 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.463036 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.464844 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.465653 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.469101 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.469450 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.469533 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.472254 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:41.472255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:41.472256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:41.472264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:41.474105 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.474614 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.474616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.474616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.474690 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.475016 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.475017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.475017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.475163 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:41.475236 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:41.475558 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:36:41.475655 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:36:41.475848 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.460971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.66:54181)
0620 18:36:41.461478 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.461524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.462850 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.464250 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.464255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.464475 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.464730 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.464735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.465855 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.465856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.468995 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.468998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.469620 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.469621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.469786 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.470486 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.470497 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.472153 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:41.474204 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.474211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.474213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.474509 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.474789 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.474790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.474791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.474925 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.475234 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:41.475239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:41.475384 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 18:36:41.475477 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:36:41.475616 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":89,"threads_started":1}
I20260620 18:36:41.476563 23242 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:41.476655 23242 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:41.476825 23242 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:41.477217 22893 ts_manager.cc:194] Registered new tserver with Master: 2bee97b1d0ae489aa6ff7522a42bcdfe (127.10.109.66:37735)
I20260620 18:36:41.477741 22893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:54181
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:36:41.501729 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:41.629014 23247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:41.629339 23247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:41.629423 23247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:41.631186 23247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:41.631389 23247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:41.631486 23247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:36:41.633579 23247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:41.634049 23247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:41.634359 23247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:41.640674 23254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.640755 23253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.640889 23256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:41.641127 23247 server_base.cc:1034] running on GCE node
I20260620 18:36:41.641309 23247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:41.641541 23247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:41.642755 23247 hybrid_clock.cc:648] HybridClock initialized: now 1781980601642713 us; error 39 us; skew 500 ppm
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:41.645854 23247 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:36:41.646373 23247 webserver.cc:460] Webserver started at http://127.10.109.67:39731/ using document root <none> and password file <none>
I20260620 18:36:41.646667 23247 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:41.646800 23247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:41.646993 23247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:41.667116 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "2f4e11e978ff41229f249bd50d37db5f"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "a0c73ba957453a1cd9a10d8eb79aecd6"
server_key_iv: "04d0218c077d2d7a061249a23e396a10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.667837 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "2f4e11e978ff41229f249bd50d37db5f"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "a0c73ba957453a1cd9a10d8eb79aecd6"
server_key_iv: "04d0218c077d2d7a061249a23e396a10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.669719 23247 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:36:41.684080 23263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.684293 23247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:41.684381 23247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "2f4e11e978ff41229f249bd50d37db5f"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "a0c73ba957453a1cd9a10d8eb79aecd6"
server_key_iv: "04d0218c077d2d7a061249a23e396a10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.684466 23247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:41.700099 23247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:41.702250 23247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:41.702442 23247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:41.702682 23247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:41.703106 23247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:41.703194 23247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.703277 23247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:41.703342 23247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.709488 23247 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:35477
I20260620 18:36:41.709928 23247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:41.712576 23376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:35477 every 8 connection(s)
I20260620 18:36:41.716750 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23247
I20260620 18:36:41.716872 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:41.717096 10677 external_mini_cluster.cc:1428] Setting key 8aed11837d6f1036f38b27a49db0c6fc
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:41.724198 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.713343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:43447 (local address 127.10.109.126:44155)
0620 18:36:41.713385 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.713387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.713842 (+ 455us) server_negotiation.cc:407] Connection header received
0620 18:36:41.714516 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.714517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.714528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.714559 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.715379 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.716112 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.718541 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.718872 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.718925 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.721566 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:41.721568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:41.721568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:41.721575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:41.722876 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.723271 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.723272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.723273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.723326 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:41.723577 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:41.723578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:41.723578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:41.723690 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:41.723764 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:41.724034 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:36:41.724109 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:36:41.724579 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.713086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.67:43447)
0620 18:36:41.713726 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.713764 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.714394 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.714729 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.714733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.715010 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.715269 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.715275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.716258 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.716259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.718441 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.718444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.719021 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.719022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.719151 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.719700 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.719716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.721458 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:41.722963 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.722969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.722971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.723182 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.723400 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:41.723401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:41.723401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:41.723509 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:41.723755 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:41.723761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:41.723903 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 18:36:41.724014 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:36:41.724152 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":82,"threads_started":1}
I20260620 18:36:41.725011 23377 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:41.725121 23377 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:41.725293 23377 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:41.725683 22893 ts_manager.cc:194] Registered new tserver with Master: 2f4e11e978ff41229f249bd50d37db5f (127.10.109.67:35477)
I20260620 18:36:41.726114 22893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:43447
I20260620 18:36:41.728469 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:41.729019 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 22974
I20260620 18:36:41.734274 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23111
I20260620 18:36:41.739706 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23247
I20260620 18:36:41.746035 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 22861
I20260620 18:36:41.752362 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44155
--webserver_interface=127.10.109.126
--webserver_port=41849
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:44155
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:41.836131 23380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:41.836311 23380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:41.836333 23380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:41.837460 23380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:41.837534 23380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:41.837560 23380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:41.837574 23380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:41.837616 23380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:41.839178 23380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:44155
--ranger_config_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=41849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:41.839395 23380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:41.839617 23380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:41.844348 23386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.844362 23385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:41.844370 23388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:41.844816 23380 server_base.cc:1034] running on GCE node
I20260620 18:36:41.845057 23380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:41.845366 23380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:41.846542 23380 hybrid_clock.cc:648] HybridClock initialized: now 1781980601846521 us; error 33 us; skew 500 ppm
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:41.849078 23380 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:41.849568 23380 webserver.cc:460] Webserver started at http://127.10.109.126:41849/ using document root <none> and password file <none>
I20260620 18:36:41.849809 23380 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:41.849859 23380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:36:41 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980601, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:41.892625 23380 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:41.893580 23396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:41.893796 23380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:41.893894 23380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "c3f64c627c8a4b7aacad45a589e56073"
format_stamp: "Formatted at 2026-06-20 18:36:40 on dist-test-slave-3jql"
server_key: "a91657072f1a6491579c72aacf213b15"
server_key_iv: "47c9d3325304a011b946a2b32d81fc2c"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:41.894480 23380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:41.914798 23380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:41.916733 23380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:41.916899 23380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:41.921332 23380 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:44155
I20260620 18:36:41.921376 23448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:44155 every 8 connection(s)
I20260620 18:36:41.921741 23380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:41.923031 23449 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:41.923692 23449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Bootstrap starting.
I20260620 18:36:41.927557 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23380
I20260620 18:36:41.927851 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:44617
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:44155
--webserver_port=44125
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:41.931764 23449 log.cc:824] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:41.933374 23449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:41.933640 23449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Bootstrap complete.
I20260620 18:36:41.936074 23449 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:41.936317 23449 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f64c627c8a4b7aacad45a589e56073, State: Initialized, Role: FOLLOWER
I20260620 18:36:41.936511 23449 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:41.936597 23449 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:41.936648 23449 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:41.936693 23449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:41.937642 23449 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:41.937773 23449 leader_election.cc:302] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f64c627c8a4b7aacad45a589e56073; no voters:
I20260620 18:36:41.937932 23449 leader_election.cc:288] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:36:41.937976 23453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:36:41.938136 23453 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [term 2 LEADER]: Becoming Leader. State: Replica: c3f64c627c8a4b7aacad45a589e56073, State: Running, Role: LEADER
I20260620 18:36:41.938176 23449 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:41.938303 23453 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } }
I20260620 18:36:41.938673 23455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f64c627c8a4b7aacad45a589e56073. Latest consensus state: current_term: 2 leader_uuid: "c3f64c627c8a4b7aacad45a589e56073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } } }
I20260620 18:36:41.938768 23455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:41.938971 23454 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3f64c627c8a4b7aacad45a589e56073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f64c627c8a4b7aacad45a589e56073" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 44155 } } }
I20260620 18:36:41.939065 23454 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:41.939180 23459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:41.939604 23459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:41.939715 23459 catalog_manager.cc:1259] Loaded cluster ID: af32d1b3134947d29b93bb1b94fceb84
I20260620 18:36:41.939747 23459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:41.939949 23449 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:36:41.943269 23459 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:41.943814 23459 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Loaded TSK: 0
I20260620 18:36:41.944087 23459 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:36:42.038940 23451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:42.039198 23451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:42.039235 23451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:42.040868 23451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:42.040973 23451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:42.041028 23451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:36:42.043133 23451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:44617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=44125
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:42.043485 23451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:42.043761 23451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:42.051095 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:42.051473 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:42.052096 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:42.053068 23451 server_base.cc:1034] running on GCE node
I20260620 18:36:42.053419 23451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:42.053702 23451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:42.054817 23451 hybrid_clock.cc:648] HybridClock initialized: now 1781980602054782 us; error 49 us; skew 500 ppm
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:42.057879 23451 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:36:42.058385 23451 webserver.cc:460] Webserver started at http://127.10.109.65:44125/ using document root <none> and password file <none>
I20260620 18:36:42.058663 23451 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:42.058773 23451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:42.083467 23451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:42.084431 23497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.084618 23451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:42.084698 23451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "815632fac60d4a7d8b00d9148443d6ee"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "8128504149beac1b5f2b9c126209e1dc"
server_key_iv: "9b03f27aa5d7d9b3d7c78b7dfa9ebd10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:42.085139 23451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:42.101796 23451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:42.105677 23451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:42.105880 23451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:42.106158 23451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:42.106534 23451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:42.106585 23451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.106616 23451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:42.106667 23451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.113778 23451 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:44617
I20260620 18:36:42.114256 23451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:42.120500 23610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:44617 every 8 connection(s)
I20260620 18:36:42.123602 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23451
I20260620 18:36:42.123898 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:37735
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:44155
--webserver_port=46371
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:42.134474 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.121417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:38441 (local address 127.10.109.126:44155)
0620 18:36:42.121771 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.121781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.121820 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:36:42.122435 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.122442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.122495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.122640 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.123323 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.124664 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.126260 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.126915 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.127067 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.129585 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:42.129588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:42.129589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:42.129600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:42.131858 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.132416 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.132420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.132420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.132532 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.132828 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.132829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.132829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.133174 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:42.133259 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:42.134117 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 18:36:42.134244 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":235,"threads_started":1}
I20260620 18:36:42.144598 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.121142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.65:38441)
0620 18:36:42.121306 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.121346 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.122216 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.122705 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.122709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.122949 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.123219 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.123225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.124785 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.124786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.126156 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.126159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.127173 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.127174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.127312 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.127848 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.127857 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.129466 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:42.131969 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.131975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.131977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.132319 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.132616 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.132617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.132617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.132750 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.133290 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:42.133318 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:42.133474 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:36:42.133589 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:36:42.133750 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260620 18:36:42.145566 23611 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:42.145651 23611 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.145819 23611 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.146636 23611 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:44155 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:36:42.146696 23611 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.146771 23611 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:42.147162 23611 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.147241 23611 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.147579 23611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:42.232686 23615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:42.232993 23615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:42.233058 23615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:42.234689 23615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:42.234788 23615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:42.234858 23615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:36:42.236861 23615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:37735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=46371
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:42.237102 23615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:42.237313 23615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:42.242581 23620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:42.242576 23623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:42.242576 23621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:42.242816 23615 server_base.cc:1034] running on GCE node
I20260620 18:36:42.243031 23615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:42.243279 23615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:42.244469 23615 hybrid_clock.cc:648] HybridClock initialized: now 1781980602244447 us; error 35 us; skew 500 ppm
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:42.246963 23615 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:36:42.247427 23615 webserver.cc:460] Webserver started at http://127.10.109.66:46371/ using document root <none> and password file <none>
I20260620 18:36:42.247646 23615 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:42.247721 23615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:42.264536 23615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 18:36:42.265537 23631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.265689 23615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:42.265761 23615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "3e8fcb9d61289569367da7d95cd4d26f"
server_key_iv: "677e244ce2ff716df26444dfdbb1a6df"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:42.266203 23615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:42.302445 23615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:42.306046 23615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:42.306216 23615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:42.306484 23615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:42.306857 23615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:42.306910 23615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.306954 23615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:42.306979 23615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.314054 23615 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37735
I20260620 18:36:42.314180 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37735 every 8 connection(s)
I20260620 18:36:42.314567 23615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:42.320366 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23615
I20260620 18:36:42.320667 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.109.84:40723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:35477
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:44155
--webserver_port=39731
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:42.334589 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.319857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:35205 (local address 127.10.109.126:44155)
0620 18:36:42.319915 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.319917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.320106 (+ 189us) server_negotiation.cc:407] Connection header received
0620 18:36:42.320793 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.320794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.320809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.320840 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.321624 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.322631 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.324232 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.324538 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.324586 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.331182 (+ 6596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:42.331183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:42.331184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:42.331191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:42.332948 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.333443 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.333444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.333444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.333502 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.333778 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.333778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.333779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.333938 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:42.333984 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:42.334384 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:36:42.334491 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:36:42.334839 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.319583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.66:35205)
0620 18:36:42.319984 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.320028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.320647 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.321028 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.321032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.321254 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.321514 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.321520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.322873 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.322874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.324127 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.324130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.324691 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.324692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.324825 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.326898 (+ 2073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.326908 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.331065 (+ 4157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:42.333046 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.333052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.333055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.333351 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.333571 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.333572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.333572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.333699 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.334037 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:42.334043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:42.334208 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:36:42.334358 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:36:42.334516 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":79,"threads_started":1}
I20260620 18:36:42.335398 23745 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:42.335512 23745 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.335733 23745 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.336498 23745 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:44155 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:36:42.336581 23745 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.336706 23745 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:42.337144 23745 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.337230 23745 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.337553 23745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:42.429817 23748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:42.430152 23748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:42.430229 23748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:42.431908 23748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:42.432004 23748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:42.432089 23748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:36:42.434185 23748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.109.84:40723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:35477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=39731
--tserver_master_addrs=127.10.109.126:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:42.434600 23748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:42.434937 23748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:42.441223 23757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:42.441227 23755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:42.442471 23748 server_base.cc:1034] running on GCE node
W20260620 18:36:42.442878 23754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:42.443128 23748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:42.443403 23748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:42.450809 23748 hybrid_clock.cc:648] HybridClock initialized: now 1781980602450792 us; error 30 us; skew 500 ppm
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:42.453292 23748 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:36:42.453796 23748 webserver.cc:460] Webserver started at http://127.10.109.67:39731/ using document root <none> and password file <none>
I20260620 18:36:42.454067 23748 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:42.454118 23748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:42.469158 23748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:42.470158 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.470474 23748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:42.470568 23748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "2f4e11e978ff41229f249bd50d37db5f"
format_stamp: "Formatted at 2026-06-20 18:36:41 on dist-test-slave-3jql"
server_key: "a0c73ba957453a1cd9a10d8eb79aecd6"
server_key_iv: "04d0218c077d2d7a061249a23e396a10"
server_key_version: "kuduclusterkey@0"
I20260620 18:36:42.471045 23748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:42.481727 23748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:42.483949 23748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:42.484129 23748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:42.484387 23748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:42.484763 23748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:42.484853 23748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.484931 23748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:42.484990 23748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:42.491860 23748 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:35477
I20260620 18:36:42.491937 23878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:35477 every 8 connection(s)
I20260620 18:36:42.492333 23748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:42.500387 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23748
I20260620 18:36:42.502442 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.493022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:38719 (local address 127.10.109.126:44155)
0620 18:36:42.493066 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.493068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.493331 (+ 263us) server_negotiation.cc:407] Connection header received
0620 18:36:42.493932 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.493933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.493943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.493996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.494608 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.495137 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.496609 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.496835 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.497035 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.499265 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:42.499267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:42.499267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:42.499274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:42.500949 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.501396 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.501397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.501397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.501459 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.501753 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.501754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.501755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.501894 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:42.501970 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:42.502269 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:36:42.502352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:36:42.502593 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.493026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:44155 (local address 127.10.109.67:38719)
0620 18:36:42.493186 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.493230 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.493828 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.494062 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.494066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.494264 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.494518 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.494524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.495225 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.495227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.496512 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.496516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.497069 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.497071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.497221 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.497707 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.497716 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.499157 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:42.501025 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.501031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.501033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.501307 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.501531 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:42.501532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:42.501532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:42.501670 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:42.501971 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:42.501977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:42.502120 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:36:42.502220 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:36:42.502364 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260620 18:36:42.503098 23879 heartbeater.cc:347] Connected to a master server at 127.10.109.126:44155
I20260620 18:36:42.503198 23879 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.503384 23879 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.504029 23879 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:44155 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:36:42.504107 23879 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.504199 23879 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:42.504570 23879 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:42.504650 23879 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
W20260620 18:36:42.504972 23879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:36:42.508514 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.501062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.10.109.126:44155)
0620 18:36:42.501176 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.501178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.501186 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:36:42.501205 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.501206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.501217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.501290 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.501999 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.502780 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.503945 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.504243 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.504320 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.505552 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:42.505553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:42.505554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:42.505560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:42.507264 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.507671 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.507672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.507673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.507732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:42.507953 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:42.507954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:42.507954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:42.508081 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:42.508256 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:42.508334 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:36:42.508418 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Jun 20 18:36:42 dist-test-slave-3jql krb5kdc[22417](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:43.147799 23611 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:43.147939 23611 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:43.148401 23413 ts_manager.cc:194] Registered new tserver with Master: 815632fac60d4a7d8b00d9148443d6ee (127.10.109.65:44617)
I20260620 18:36:43.149112 23413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:38441
Jun 20 18:36:43 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:43.337741 23745 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:43.337894 23745 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:43.338228 23413 ts_manager.cc:194] Registered new tserver with Master: 2bee97b1d0ae489aa6ff7522a42bcdfe (127.10.109.66:37735)
I20260620 18:36:43.338809 23413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:35205
I20260620 18:36:43.505204 23879 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:43.505357 23879 heartbeater.cc:511] Master 127.10.109.126:44155 requested a full tablet report, sending...
I20260620 18:36:43.505683 23413 ts_manager.cc:194] Registered new tserver with Master: 2f4e11e978ff41229f249bd50d37db5f (127.10.109.67:35477)
I20260620 18:36:43.506253 23413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:38719
I20260620 18:36:43.510473 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:43.523461 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.511436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.10.109.126:44155)
0620 18:36:43.511672 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.511674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.511828 (+ 154us) server_negotiation.cc:407] Connection header received
0620 18:36:43.511849 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.511850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.511861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.511915 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:43.512601 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.513482 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.514733 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.515075 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.515134 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.520382 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:43.520384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:43.520384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:43.520392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:43.522184 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:43.522612 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:43.522613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:43.522614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:43.522678 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:43.522919 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:43.522920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:43.522920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:43.523050 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:43.523155 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.523264 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.523347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20260620 18:36:43.525527 23413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:43.527521 23413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:43.539145 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.530432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37735 (local address 127.0.0.1:42364)
0620 18:36:43.534213 (+ 3781us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.534222 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.534257 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.535021 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.535022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.535024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:43.535075 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.535076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.536278 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.536279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.537408 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.537411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.538798 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.538799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.538884 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.538940 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:36:43.539008 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:36:43.539059 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3766,"thread_start_us":99,"threads_started":1}
I20260620 18:36:43.539621 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.531171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.10.109.66:37735)
0620 18:36:43.534794 (+ 3623us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.534798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.534808 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:36:43.534840 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.534842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.534857 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.534936 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:43.535150 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.536179 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.537510 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.538593 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.538655 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.538719 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.539504 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.539545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3594,"thread_start_us":56,"threads_started":1}
I20260620 18:36:43.540153 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.530429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:44617 (local address 127.0.0.1:47442)
0620 18:36:43.534007 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.534046 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.534113 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.534547 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.534551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.534553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:43.534624 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.534626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.535846 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.535848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.537130 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.537134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.539874 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.539875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.539949 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.539953 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:36:43.540051 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:36:43.540087 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3546,"thread_start_us":94,"threads_started":1}
I20260620 18:36:43.540570 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.531023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.10.109.65:44617)
0620 18:36:43.534360 (+ 3337us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.534364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.534375 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:36:43.534411 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.534414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.534430 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.534520 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:43.534727 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.535753 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.538424 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.539209 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.539275 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.539325 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.540452 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.540495 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3305,"thread_start_us":57,"threads_started":1}
I20260620 18:36:43.540930 23679 tablet_service.cc:1467] Processing CreateTablet for tablet 72c31ff95c034abcaa18683e4d525113 (DEFAULT_TABLE table=test-table [id=5145351bac9046f7a3928c4890f9cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:43.541132 23545 tablet_service.cc:1467] Processing CreateTablet for tablet 72c31ff95c034abcaa18683e4d525113 (DEFAULT_TABLE table=test-table [id=5145351bac9046f7a3928c4890f9cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:43.541322 23679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c31ff95c034abcaa18683e4d525113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:43.541513 23545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c31ff95c034abcaa18683e4d525113. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:36:43 dist-test-slave-3jql krb5kdc[22417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980602, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for HTTP/127.10.109.84@KRBTEST.COM
I20260620 18:36:43.545806 23913 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Bootstrap starting.
I20260620 18:36:43.548497 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.530655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:35477 (local address 127.0.0.1:51458)
0620 18:36:43.534621 (+ 3966us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.534631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.534672 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.538528 (+ 3856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.538529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.538531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:43.538580 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.538581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.546060 (+ 7479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.546061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.547204 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.547207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.548247 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.548248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.548319 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.548323 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:36:43.548374 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:36:43.548411 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3949,"thread_start_us":69,"threads_started":1}
I20260620 18:36:43.549150 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.531310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.10.109.67:35477)
0620 18:36:43.538364 (+ 7054us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.538368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.538380 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:36:43.538413 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.538415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.538430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.538518 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:43.538674 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.545964 (+ 7290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.547315 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.548086 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.548159 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.548211 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.549016 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.549065 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7021,"thread_start_us":50,"threads_started":1}
I20260620 18:36:43.550657 23813 tablet_service.cc:1467] Processing CreateTablet for tablet 72c31ff95c034abcaa18683e4d525113 (DEFAULT_TABLE table=test-table [id=5145351bac9046f7a3928c4890f9cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:43.550989 23813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c31ff95c034abcaa18683e4d525113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:43.553640 23915 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Bootstrap starting.
I20260620 18:36:43.553959 23913 tablet_bootstrap.cc:654] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:43.554306 23913 log.cc:824] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:43.554687 23914 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Bootstrap starting.
I20260620 18:36:43.555348 23914 tablet_bootstrap.cc:654] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:43.555723 23914 log.cc:824] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:43.556273 23915 tablet_bootstrap.cc:654] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:43.556615 23915 log.cc:824] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:43.557413 23913 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: No bootstrap required, opened a new log
I20260620 18:36:43.557482 23913 ts_tablet_manager.cc:1397] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260620 18:36:43.560037 23913 raft_consensus.cc:348] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.560192 23913 raft_consensus.cc:374] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:43.560225 23913 raft_consensus.cc:729] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bee97b1d0ae489aa6ff7522a42bcdfe, State: Initialized, Role: FOLLOWER
I20260620 18:36:43.560340 23913 consensus_queue.cc:260] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.560585 23913 ts_tablet_manager.cc:1428] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:43.560644 23745 heartbeater.cc:503] Master 127.10.109.126:44155 was elected leader, sending a full tablet report...
I20260620 18:36:43.561328 23915 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: No bootstrap required, opened a new log
I20260620 18:36:43.561460 23915 ts_tablet_manager.cc:1397] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 18:36:43.563967 23915 raft_consensus.cc:348] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.564177 23915 raft_consensus.cc:374] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:43.564249 23915 raft_consensus.cc:729] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4e11e978ff41229f249bd50d37db5f, State: Initialized, Role: FOLLOWER
I20260620 18:36:43.564435 23915 consensus_queue.cc:260] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.564756 23915 ts_tablet_manager.cc:1428] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:43.564901 23879 heartbeater.cc:503] Master 127.10.109.126:44155 was elected leader, sending a full tablet report...
W20260620 18:36:43.565752 23746 tablet.cc:2369] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:43.566064 23914 tablet_bootstrap.cc:492] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: No bootstrap required, opened a new log
I20260620 18:36:43.566133 23914 ts_tablet_manager.cc:1397] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260620 18:36:43.568665 23914 raft_consensus.cc:348] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.568804 23914 raft_consensus.cc:374] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:43.568831 23914 raft_consensus.cc:729] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815632fac60d4a7d8b00d9148443d6ee, State: Initialized, Role: FOLLOWER
I20260620 18:36:43.568938 23914 consensus_queue.cc:260] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.569157 23914 ts_tablet_manager.cc:1428] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:43.569216 23611 heartbeater.cc:503] Master 127.10.109.126:44155 was elected leader, sending a full tablet report...
W20260620 18:36:43.622709 23612 tablet.cc:2369] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:43.743728 23880 tablet.cc:2369] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:43.836313 23919 raft_consensus.cc:482] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:43.836463 23919 raft_consensus.cc:504] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.836853 23919 leader_election.cc:288] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 815632fac60d4a7d8b00d9148443d6ee (127.10.109.65:44617), 2f4e11e978ff41229f249bd50d37db5f (127.10.109.67:35477)
I20260620 18:36:43.840198 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.837116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:44653 (local address 127.10.109.65:44617)
0620 18:36:43.837162 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.837164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.837326 (+ 162us) server_negotiation.cc:407] Connection header received
0620 18:36:43.837410 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.837411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.837425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.837484 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:43.837656 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.838181 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839401 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.839801 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.839919 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.840087 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.840129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:36:43.840229 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.837119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:37309 (local address 127.10.109.67:35477)
0620 18:36:43.837181 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.837183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.837325 (+ 142us) server_negotiation.cc:407] Connection header received
0620 18:36:43.837408 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.837409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.837419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.837493 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:43.837654 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.838251 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839444 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.839826 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839911 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.839919 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.840116 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.840159 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:36:43.840224 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.837045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:44617 (local address 127.10.109.66:44653)
0620 18:36:43.837241 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.837251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.837342 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.837487 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.837488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.837490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:43.837579 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.837581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.838257 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.838258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.839319 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.839321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.839948 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.840024 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.840029 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:43.840082 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:36:43.840122 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":51,"thread_start_us":130,"threads_started":1}
I20260620 18:36:43.840236 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.837052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:35477 (local address 127.10.109.66:37309)
0620 18:36:43.837249 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.837258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.837341 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.837497 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.837498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.837500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:43.837585 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.837586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.838330 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.838331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.839357 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.839360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.839958 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.839959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.840025 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.840028 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:36:43.840088 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:36:43.840132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":121,"threads_started":1}
I20260620 18:36:43.840600 23833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c31ff95c034abcaa18683e4d525113" candidate_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4e11e978ff41229f249bd50d37db5f" is_pre_election: true
I20260620 18:36:43.840765 23833 raft_consensus.cc:2454] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bee97b1d0ae489aa6ff7522a42bcdfe in term 0.
I20260620 18:36:43.840817 23558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c31ff95c034abcaa18683e4d525113" candidate_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815632fac60d4a7d8b00d9148443d6ee" is_pre_election: true
I20260620 18:36:43.840965 23558 raft_consensus.cc:2454] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bee97b1d0ae489aa6ff7522a42bcdfe in term 0.
I20260620 18:36:43.841002 23634 leader_election.cc:302] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bee97b1d0ae489aa6ff7522a42bcdfe, 2f4e11e978ff41229f249bd50d37db5f; no voters:
I20260620 18:36:43.841115 23919 raft_consensus.cc:2781] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:43.841189 23919 raft_consensus.cc:482] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:43.841221 23919 raft_consensus.cc:3037] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:43.841904 23919 raft_consensus.cc:504] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.842047 23919 leader_election.cc:288] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [CANDIDATE]: Term 1 election: Requested vote from peers 815632fac60d4a7d8b00d9148443d6ee (127.10.109.65:44617), 2f4e11e978ff41229f249bd50d37db5f (127.10.109.67:35477)
I20260620 18:36:43.842233 23558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c31ff95c034abcaa18683e4d525113" candidate_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815632fac60d4a7d8b00d9148443d6ee"
I20260620 18:36:43.842238 23833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c31ff95c034abcaa18683e4d525113" candidate_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4e11e978ff41229f249bd50d37db5f"
I20260620 18:36:43.842303 23833 raft_consensus.cc:3037] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:43.842321 23558 raft_consensus.cc:3037] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:43.843055 23558 raft_consensus.cc:2454] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bee97b1d0ae489aa6ff7522a42bcdfe in term 1.
I20260620 18:36:43.843081 23833 raft_consensus.cc:2454] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bee97b1d0ae489aa6ff7522a42bcdfe in term 1.
I20260620 18:36:43.843248 23633 leader_election.cc:302] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bee97b1d0ae489aa6ff7522a42bcdfe, 815632fac60d4a7d8b00d9148443d6ee; no voters:
I20260620 18:36:43.843338 23919 raft_consensus.cc:2781] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:43.843470 23919 raft_consensus.cc:686] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 1 LEADER]: Becoming Leader. State: Replica: 2bee97b1d0ae489aa6ff7522a42bcdfe, State: Running, Role: LEADER
I20260620 18:36:43.843567 23919 consensus_queue.cc:237] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } }
I20260620 18:36:43.844348 23413 catalog_manager.cc:5495] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bee97b1d0ae489aa6ff7522a42bcdfe (127.10.109.66). New cstate: current_term: 1 leader_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bee97b1d0ae489aa6ff7522a42bcdfe" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:43.900444 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.897971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.10.109.66:37735)
0620 18:36:43.898038 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.898040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.898048 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:36:43.898083 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.898084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.898101 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.898161 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:43.898309 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.898985 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.899749 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.899932 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.899995 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.900174 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:43.900267 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:43.900325 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:43.900368 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 18:36:43.900385 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:36:43.902266 23558 raft_consensus.cc:1261] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 1 FOLLOWER]: Refusing update from remote peer 2bee97b1d0ae489aa6ff7522a42bcdfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:43.902266 23833 raft_consensus.cc:1261] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 1 FOLLOWER]: Refusing update from remote peer 2bee97b1d0ae489aa6ff7522a42bcdfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:43.902628 23932 consensus_queue.cc:1035] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "815632fac60d4a7d8b00d9148443d6ee" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:43.902781 23919 consensus_queue.cc:1035] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4e11e978ff41229f249bd50d37db5f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:43.905347 23936 mvcc.cc:205] Tried to move back new op lower bound from 7298992553580937216 to 7298992553343709184. Current Snapshot: MvccSnapshot[applied={T|T < 7298992553580937216}]
I20260620 18:36:43.907809 23413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:43.907902 23413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:43.908561 23413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f64c627c8a4b7aacad45a589e56073: Sending DeleteTablet for 3 replicas of tablet 72c31ff95c034abcaa18683e4d525113
I20260620 18:36:43.908923 23545 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c31ff95c034abcaa18683e4d525113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:43 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:47442
I20260620 18:36:43.908936 23679 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c31ff95c034abcaa18683e4d525113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:43 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:42364
I20260620 18:36:43.908936 23813 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c31ff95c034abcaa18683e4d525113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:43 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:51458
I20260620 18:36:43.909250 23945 tablet_replica.cc:330] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: stopping tablet replica
I20260620 18:36:43.909400 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23451
I20260620 18:36:43.909416 23947 tablet_replica.cc:330] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee: stopping tablet replica
I20260620 18:36:43.909511 23947 raft_consensus.cc:2229] T 72c31ff95c034abcaa18683e4d525113 P 815632fac60d4a7d8b00d9148443d6ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:43.909669 23945 raft_consensus.cc:2229] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:43.909943 23945 raft_consensus.cc:2258] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:43.910250 23946 tablet_replica.cc:330] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: stopping tablet replica
I20260620 18:36:43.910547 23945 ts_tablet_manager.cc:1905] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:43.912245 23946 raft_consensus.cc:2229] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:43.912307 23945 ts_tablet_manager.cc:1918] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:43.912415 23945 log.cc:1196] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/72c31ff95c034abcaa18683e4d525113
I20260620 18:36:43.912431 23946 raft_consensus.cc:2258] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:43.912727 23945 ts_tablet_manager.cc:1939] T 72c31ff95c034abcaa18683e4d525113 P 2bee97b1d0ae489aa6ff7522a42bcdfe: Deleting consensus metadata
I20260620 18:36:43.913040 23946 ts_tablet_manager.cc:1905] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:43.913156 23399 catalog_manager.cc:4841] TS 2bee97b1d0ae489aa6ff7522a42bcdfe (127.10.109.66:37735): tablet 72c31ff95c034abcaa18683e4d525113 (table test-table [id=5145351bac9046f7a3928c4890f9cb4f]) successfully deleted
I20260620 18:36:43.914657 23946 ts_tablet_manager.cc:1918] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:43.914732 23946 log.cc:1196] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/72c31ff95c034abcaa18683e4d525113
I20260620 18:36:43.915073 23946 ts_tablet_manager.cc:1939] T 72c31ff95c034abcaa18683e4d525113 P 2f4e11e978ff41229f249bd50d37db5f: Deleting consensus metadata
I20260620 18:36:43.915486 23399 catalog_manager.cc:4841] TS 2f4e11e978ff41229f249bd50d37db5f (127.10.109.67:35477): tablet 72c31ff95c034abcaa18683e4d525113 (table test-table [id=5145351bac9046f7a3928c4890f9cb4f]) successfully deleted
W20260620 18:36:43.915853 23398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:44617 (error 108)
I20260620 18:36:43.916347 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.916243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:44617 (local address 127.0.0.1:47456)
0620 18:36:43.916280 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.916312 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 18:36:43.916455 23398 catalog_manager.cc:4584] TS 815632fac60d4a7d8b00d9148443d6ee (127.10.109.65:44617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72c31ff95c034abcaa18683e4d525113: Network error: Client connection negotiation failed: client connection to 127.10.109.65:44617: connect: Connection refused (error 111)
I20260620 18:36:43.916616 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23615
I20260620 18:36:43.921043 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23748
I20260620 18:36:43.925489 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23380
I20260620 18:36:43.930989 10677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:36:44.265483 10677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:36:44.265642 10677 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:36:44.600461 10677 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:36:44.600 UTC [22447] LOG: received smart shutdown request
2026-06-20 18:36:44.604 UTC [22447] LOG: background worker "logical replication launcher" (PID 22456) exited with exit code 1
2026-06-20 18:36:44.604 UTC [22451] LOG: shutting down
2026-06-20 18:36:44.784 UTC [22447] LOG: database system is shut down
2026-06-20T18:36:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47866 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:44 dist-test-slave-3jql krb5kdc[23957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:44 dist-test-slave-3jql krb5kdc[23957](info): set up 2 sockets
Jun 20 18:36:44 dist-test-slave-3jql krb5kdc[23957](info): commencing operation
krb5kdc: starting...
W20260620 18:36:45.943125 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:36:46.058454 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40687
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:46.136364 23970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:46.136548 23970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:46.136569 23970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:46.137665 23970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:46.137704 23970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:46.137718 23970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:46.137758 23970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:46.137784 23970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:46.139184 23970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:46.139436 23970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:46.139650 23970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:46.144476 23975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:46.144423 23976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:46.144454 23970 server_base.cc:1034] running on GCE node
W20260620 18:36:46.144423 23978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:46.144857 23970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:46.145152 23970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:46.146312 23970 hybrid_clock.cc:648] HybridClock initialized: now 1781980606146295 us; error 44 us; skew 500 ppm
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:46.148618 23970 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:46.149075 23970 webserver.cc:460] Webserver started at http://127.10.109.126:33437/ using document root <none> and password file <none>
I20260620 18:36:46.149282 23970 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:46.149345 23970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:46.149484 23970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:46.150449 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "54e45df48069478fa8be0365038eec68"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "886690ed32ea4c2001182ae8b6c335fc"
server_key_iv: "d4fb0a9895d9f7257f9ffeed2f76b0b2"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.150856 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "54e45df48069478fa8be0365038eec68"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "886690ed32ea4c2001182ae8b6c335fc"
server_key_iv: "d4fb0a9895d9f7257f9ffeed2f76b0b2"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.152154 23970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:46.152859 23985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:46.153036 23970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:36:46.153127 23970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "54e45df48069478fa8be0365038eec68"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "886690ed32ea4c2001182ae8b6c335fc"
server_key_iv: "d4fb0a9895d9f7257f9ffeed2f76b0b2"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.153208 23970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:46.160769 23970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:46.162503 23970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:46.162644 23970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:46.166386 23970 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:40687
I20260620 18:36:46.166438 24037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:40687 every 8 connection(s)
I20260620 18:36:46.166785 23970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:46.167544 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:46.168946 24038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.172443 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 23970
I20260620 18:36:46.172546 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:36:46.172755 10677 external_mini_cluster.cc:1428] Setting key a24cbac718c0660a2b3200c29ce91fd6
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:46.178527 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.169562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.10.109.126:40687)
0620 18:36:46.169709 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.169716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.169812 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:36:46.170280 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.170282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.170303 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.170379 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.170944 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.171739 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.172924 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.173228 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.173306 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.175568 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.175574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.175574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.175580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.177037 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.177501 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.177502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.177503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.177573 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.177819 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.177820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.177820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.177998 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.178056 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.178276 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.178377 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260620 18:36:46.178637 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.169531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40687 (local address 127.0.0.1:53628)
0620 18:36:46.169675 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.169710 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.170169 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.170392 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.170394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.170608 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.170862 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.170866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.171836 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.171837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.172825 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.172828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.173319 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.173320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.173408 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.173837 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.173847 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.175477 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.177121 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.177129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.177131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.177420 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.177637 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.177638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.177639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.177753 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.178063 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.178070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.178145 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.178256 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.178473 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":60,"thread_start_us":77,"threads_started":1}
I20260620 18:36:46.179309 24038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.179636 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.179516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43472)
0620 18:36:46.179565 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.179610 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:36:46.179769 23989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
I20260620 18:36:46.180208 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.180100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43488)
0620 18:36:46.180148 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.180179 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:36:46.180326 24038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:33775: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.109.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
I20260620 18:36:46.197587 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:33775
--webserver_interface=127.10.109.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:46.229564 24038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } attempt: 1
I20260620 18:36:46.230068 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.229914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43496)
0620 18:36:46.229982 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.230029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:36:46.230409 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.230324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43498)
0620 18:36:46.230365 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.230390 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:36:46.230516 24038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:33775: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
W20260620 18:36:46.275641 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:46.275815 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:46.275835 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:46.276918 24045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:46.276957 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:46.276970 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:46.277014 24045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:46.277038 24045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:46.278447 24045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.109.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:46.278668 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:46.278880 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:46.283301 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:46.283419 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:46.283545 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:46.283555 24045 server_base.cc:1034] running on GCE node
I20260620 18:36:46.283985 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:46.284246 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:46.285403 24045 hybrid_clock.cc:648] HybridClock initialized: now 1781980606285395 us; error 38 us; skew 500 ppm
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:46.287787 24045 init.cc:377] Logged in from keytab as kudu/127.10.109.125@KRBTEST.COM (short username kudu)
I20260620 18:36:46.288204 24045 webserver.cc:460] Webserver started at http://127.10.109.125:39823/ using document root <none> and password file <none>
I20260620 18:36:46.288404 24045 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:46.288446 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:46.288596 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:46.289565 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data/instance:
uuid: "bfd982374a2c44a798b699a97a2abfc9"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "2d4c67073499241ace0983dcbde9f50c"
server_key_iv: "4f577d5335c0f6f8c439e0260162f7b7"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.289891 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal/instance:
uuid: "bfd982374a2c44a798b699a97a2abfc9"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "2d4c67073499241ace0983dcbde9f50c"
server_key_iv: "4f577d5335c0f6f8c439e0260162f7b7"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.291205 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:36:46.291900 24060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:46.292101 24045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:46.292196 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "bfd982374a2c44a798b699a97a2abfc9"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "2d4c67073499241ace0983dcbde9f50c"
server_key_iv: "4f577d5335c0f6f8c439e0260162f7b7"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.292264 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:46.298691 24038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } attempt: 2
I20260620 18:36:46.299125 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.299002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43508)
0620 18:36:46.299057 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.299093 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:36:46.299453 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.299368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43524)
0620 18:36:46.299415 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.299437 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:36:46.299563 24038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:33775: Network error: Client connection negotiation failed: client connection to 127.10.109.125:33775: connect: Connection refused (error 111)
I20260620 18:36:46.315619 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:46.317394 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:46.317543 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:46.321449 24045 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.125:33775
I20260620 18:36:46.321480 24112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.125:33775 every 8 connection(s)
I20260620 18:36:46.321802 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
I20260620 18:36:46.322561 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:46.324070 24113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:46.331880 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24045
I20260620 18:36:46.331993 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-1/wal/instance
I20260620 18:36:46.332193 10677 external_mini_cluster.cc:1428] Setting key 07664d2d1eb30e30e423a9f697c3df26
I20260620 18:36:46.332263 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.324690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.10.109.126:40687)
0620 18:36:46.324777 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.324779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.324925 (+ 146us) server_negotiation.cc:407] Connection header received
0620 18:36:46.325331 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.325332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.325343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.325395 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.325956 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.326433 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.327379 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.327582 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.327650 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.329552 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.329553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.329553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.329559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.330719 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.331188 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.331189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.331189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.331243 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.331560 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.331560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.331560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.331662 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.331815 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.332074 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.332164 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:36:46.332403 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.324670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40687 (local address 127.0.0.1:53630)
0620 18:36:46.324810 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.324850 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.325258 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.325451 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.325455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.325674 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.325888 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.325892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.326508 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.326508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.327298 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.327301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.327682 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.327683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.327788 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.328201 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.328209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.329472 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.330917 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.330923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.330924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.331125 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.331353 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.331354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.331354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.331499 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.331801 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.331806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.331923 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.332028 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.332171 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260620 18:36:46.333030 24113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.124@KRBTEST.COM; defaulting to no policy
I20260620 18:36:46.340708 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.333258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43526)
0620 18:36:46.333301 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.333310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.333349 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.333890 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.333891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.334045 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.334101 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.334103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.334978 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.334979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.335858 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.335861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.336278 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.336279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.336369 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.336701 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.336706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.338043 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.339831 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.339832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.339833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.340054 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.340247 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.340248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.340249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.340305 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.340503 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.340503 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.340535 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.340584 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.340654 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 18:36:46.340916 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.333620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.10.109.125:33775)
0620 18:36:46.333761 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.333765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.333778 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:36:46.333795 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.333797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.333807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.333871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.334170 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.334900 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.335936 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.336204 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.336261 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.338135 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.338141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.338141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.338147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.339710 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.340136 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.340138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.340139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.340189 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.340349 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.340349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.340349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.340454 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.340506 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.340603 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.340804 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 18:36:46.341414 24113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.341689 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.341597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48096)
0620 18:36:46.341634 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.341671 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 18:36:46.341838 24063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
I20260620 18:36:46.342254 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.342171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48104)
0620 18:36:46.342210 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.342239 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 18:36:46.342370 24113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:33341: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
I20260620 18:36:46.357553 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:33341
--webserver_interface=127.10.109.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:46.390756 24038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } attempt: 3
I20260620 18:36:46.391573 24113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } attempt: 1
I20260620 18:36:46.391980 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.391864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48116)
0620 18:36:46.391911 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.391941 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:36:46.392360 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.392253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48126)
0620 18:36:46.392302 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.392338 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:36:46.392500 24113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:33341: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:36:46.398068 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.391194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.10.109.125:33775)
0620 18:36:46.391270 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.391272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.391284 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:36:46.391353 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.391353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.391368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.391438 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.392090 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.392782 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.393449 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.393653 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.393726 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.395765 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.395766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.395767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.395772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.397052 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.397429 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.397430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.397430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.397477 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.397655 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.397656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.397656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.397746 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.397794 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.397960 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.398017 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:36:46.398108 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.391105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43536)
0620 18:36:46.391174 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.391186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.391230 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.391500 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.391502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.391819 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.391944 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.391947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.392861 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.392863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.393349 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.393350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.393724 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.393725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.393804 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.394232 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.394237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.395695 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.397131 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.397132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.397132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.397362 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.397533 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.397534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.397534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.397599 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.397801 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.397801 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.397850 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.397924 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.398018 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:36:46.398443 24038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.398733 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.398648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48138)
0620 18:36:46.398682 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.398707 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 18:36:46.399070 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.398982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48142)
0620 18:36:46.399024 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.399047 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:36:46.399155 24038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:33341: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
I20260620 18:36:46.430295 24038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } attempt: 1
I20260620 18:36:46.430646 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.430540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48154)
0620 18:36:46.430590 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.430621 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:36:46.431007 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.430918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48170)
0620 18:36:46.430961 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.430990 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:36:46.431089 24038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:33341: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
W20260620 18:36:46.438472 24120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:46.438642 24120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:46.438663 24120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:46.439766 24120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:46.439807 24120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:46.439829 24120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:46.439842 24120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:46.439854 24120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:46.441345 24120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:40687,127.10.109.125:33775,127.10.109.124:33341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.109.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:46.441563 24120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:46.441771 24120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:46.446422 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:46.446384 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:46.446429 24120 server_base.cc:1034] running on GCE node
W20260620 18:36:46.446388 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:46.446921 24120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:46.447201 24120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:46.448361 24120 hybrid_clock.cc:648] HybridClock initialized: now 1781980606448332 us; error 39 us; skew 500 ppm
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:46.450799 24120 init.cc:377] Logged in from keytab as kudu/127.10.109.124@KRBTEST.COM (short username kudu)
I20260620 18:36:46.451246 24120 webserver.cc:460] Webserver started at http://127.10.109.124:38345/ using document root <none> and password file <none>
I20260620 18:36:46.451470 24120 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:46.451515 24120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:46.451655 24120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:46.452665 24120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data/instance:
uuid: "9a4ed520cc324fc59529f79d1cb8be3f"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "a60be6752f6b94ccbc101c49a05eb64f"
server_key_iv: "c7e0bed40586b21299bf33fcbf1f0f3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.453016 24120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal/instance:
uuid: "9a4ed520cc324fc59529f79d1cb8be3f"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "a60be6752f6b94ccbc101c49a05eb64f"
server_key_iv: "c7e0bed40586b21299bf33fcbf1f0f3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.454265 24120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:36:46.455018 24135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:46.455174 24120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:46.455273 24120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "9a4ed520cc324fc59529f79d1cb8be3f"
format_stamp: "Formatted at 2026-06-20 18:36:46 on dist-test-slave-3jql"
server_key: "a60be6752f6b94ccbc101c49a05eb64f"
server_key_iv: "c7e0bed40586b21299bf33fcbf1f0f3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:46.455325 24120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:46.460632 24113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } attempt: 2
I20260620 18:36:46.461005 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.460859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48182)
0620 18:36:46.460944 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.460978 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:36:46.461352 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.461281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48188)
0620 18:36:46.461317 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.461338 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 18:36:46.461445 24113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:33341: Network error: Client connection negotiation failed: client connection to 127.10.109.124:33341: connect: Connection refused (error 111)
I20260620 18:36:46.482713 24120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:46.484490 24120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:46.484592 24120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:46.488250 24120 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.124:33341
I20260620 18:36:46.488273 24187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.124:33341 every 8 connection(s)
I20260620 18:36:46.488608 24120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
I20260620 18:36:46.489442 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:46.490993 24188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.491583 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24120
I20260620 18:36:46.491698 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980452801806-10677-0/minicluster-data/master-2/wal/instance
I20260620 18:36:46.491922 10677 external_mini_cluster.cc:1428] Setting key 8c21cc5f0541bee6963a36638a749c65
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:36:46.500120 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.491684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.10.109.126:40687)
0620 18:36:46.491748 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.491750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.491997 (+ 247us) server_negotiation.cc:407] Connection header received
0620 18:36:46.492628 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.492629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.492639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.492692 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.493265 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.493733 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.494644 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.494888 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.494974 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.497126 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.497127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.497127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.497134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.498773 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.499125 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.499127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.499127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.499185 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.499414 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.499415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.499416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.499537 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.499587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.499934 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.500033 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:36:46.500226 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.491654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:40687 (local address 127.0.0.1:53640)
0620 18:36:46.491854 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.491902 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.492531 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.492732 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.492736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.493036 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.493175 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.493179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.493815 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.493816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.494504 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.494506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.494981 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.494983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.495103 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.495552 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.495559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.497044 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.498847 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.498852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.498854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.499054 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.499268 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.499268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.499269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.499358 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.499605 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.499611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.499761 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.499912 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.500060 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260620 18:36:46.500779 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.492289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.10.109.124:33341)
0620 18:36:46.492607 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.492611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.492640 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:36:46.492726 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.492728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.492746 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.492827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.493391 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.494185 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.495019 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.495310 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.495392 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.497671 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.497678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.497679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.497686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.499370 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.499710 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.499718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.499718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.499763 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.499977 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.499978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.499978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.500133 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.500506 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.500554 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.500635 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":290,"thread_start_us":67,"threads_started":1}
I20260620 18:36:46.500753 24188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:46.506242 24038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } attempt: 2
I20260620 18:36:46.507439 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.501006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:33775 (local address 127.0.0.1:43550)
0620 18:36:46.501043 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.501051 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.501084 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.501214 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.501215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.501340 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.501382 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.501383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.502094 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.502094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.502561 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.502562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.502944 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.502945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.503012 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.503330 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.503335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.504545 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.506259 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.506260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.506260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.506473 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.506809 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.506810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.506811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.506872 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.507117 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.507118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.507162 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.507233 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.507320 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 18:36:46.507450 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.501055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.10.109.125:33775)
0620 18:36:46.501093 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.501095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.501104 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:36:46.501128 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.501128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.501138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.501198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.501460 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.502016 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.502639 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.502883 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.502933 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.504625 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.504626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.504626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.504633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.506177 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.506572 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.506573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.506574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.506630 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.506943 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.506944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.506944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.507051 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.507125 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.507282 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.507356 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:36:46.508195 24188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:46.509126 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.501446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.10.109.126:40687)
0620 18:36:46.501487 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.501489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.501534 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:36:46.501589 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.501589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.501598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.501650 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.502033 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.502498 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.503371 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.503538 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.503585 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.505798 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.505799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.505800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.505806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.507883 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.508256 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.508257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.508258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.508309 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.508494 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.508494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.508495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.508599 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.508750 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.508945 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.509023 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:36:46.514048 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.506494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48210)
0620 18:36:46.506538 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.506548 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.506584 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.506700 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.506701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.506889 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.506928 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.506930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.507711 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.507718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.508397 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.508399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.508861 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.508862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.508920 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.509310 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.509314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.510565 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.512966 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.512967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.512967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.513161 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.513371 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.513372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.513373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.513442 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.513719 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.513720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.513781 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.513833 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.513937 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:36:46.514461 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.506527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.10.109.124:33341)
0620 18:36:46.506565 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.506567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.506575 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:36:46.506639 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.506640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.506650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.506697 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.507010 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.507632 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.508478 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.508774 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.509209 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.510648 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.510649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.510650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.510657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.512887 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.513240 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.513241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.513241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.513299 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.513506 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.513507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.513507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.513635 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.514250 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.514290 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.514412 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:36:46.515671 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.508467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48222)
0620 18:36:46.508623 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.508633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.508680 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.508803 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.508804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.508941 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.508970 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.508971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.509866 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.509867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.510395 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.510397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.510817 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.510818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.510865 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.511330 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.511334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.512512 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.514616 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.514617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.514617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.514870 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.515076 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.515077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.515077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.515138 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.515381 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.515382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.515417 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.515480 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.515568 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 18:36:46.515909 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.508489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.10.109.124:33341)
0620 18:36:46.508635 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.508636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.508666 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:36:46.508726 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.508727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.508735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.508801 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.509053 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.509782 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.510473 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.510706 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.510816 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.512585 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.512586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.512586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.512592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.514527 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.514938 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.514939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.514939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.514999 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.515192 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.515193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.515194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.515316 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.515388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.515506 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.515615 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":128,"thread_start_us":107,"threads_started":1}
I20260620 18:36:46.516314 24038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: Bootstrap starting.
I20260620 18:36:46.517000 24038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:46.517100 24188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: Bootstrap starting.
I20260620 18:36:46.517309 24038 log.cc:824] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:46.517724 24188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:46.518031 24188 log.cc:824] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:46.518183 24038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: No bootstrap required, opened a new log
I20260620 18:36:46.518777 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.509930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.10.109.125:33775)
0620 18:36:46.509962 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.509963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.510392 (+ 429us) server_negotiation.cc:407] Connection header received
0620 18:36:46.510447 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.510448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.510459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.510511 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.511633 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.512313 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.513255 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.513521 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.513995 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.516113 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.516115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.516115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.516122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.517485 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.517876 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.517877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.517877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.517928 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.518121 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.518122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.518122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.518253 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.518407 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.518587 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.518659 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:36:46.518811 24188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: No bootstrap required, opened a new log
I20260620 18:36:46.520344 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:46.520303 24038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.520442 24038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:46.520515 24038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e45df48069478fa8be0365038eec68, State: Initialized, Role: FOLLOWER
I20260620 18:36:46.520684 24038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.520670 24188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.520829 24188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:46.520888 24188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4ed520cc324fc59529f79d1cb8be3f, State: Initialized, Role: FOLLOWER
I20260620 18:36:46.521015 24038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:46.521064 24188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.521013 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.521163 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:46.521385 24188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:46.523765 23970 master_runner.cc:186] Error getting master registration for 127.10.109.125:33775: OK, instance_id { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" instance_seqno: 1781980606320288 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:36:46.526764 23970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:36:46.524385 24120 master_runner.cc:186] Error getting master registration for 127.10.109.125:33775: OK, instance_id { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" instance_seqno: 1781980606320288 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:36:46.527091 24120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:36:46.527753 24222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:46.527840 24222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:36:46.529933 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.522179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.10.109.124:33341)
0620 18:36:46.522212 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.522214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.522221 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:36:46.522240 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.522241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.522251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.522296 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.523175 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.523913 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.524910 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.525208 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.525247 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.527028 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.527029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.527029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.527035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.528641 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.529047 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.529048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.529048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.529095 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.529295 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.529296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.529296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.529440 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.529529 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.529761 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.529846 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 18:36:46.530223 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.522142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.10.109.126:40687)
0620 18:36:46.522892 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.522894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.522903 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:36:46.522976 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.522977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.523037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.523095 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.524050 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.524739 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.525687 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.525970 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.526033 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.527388 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.527389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.527390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.527397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.528939 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.529324 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.529325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.529326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.529369 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.529567 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.529568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.529568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.529686 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.530026 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.530075 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.530144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728}
I20260620 18:36:46.531265 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.531370 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:36:46.531813 24226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:46.531903 24226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:36:46.535022 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.522336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.10.109.125:33775)
0620 18:36:46.522369 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.522371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.528160 (+ 5789us) server_negotiation.cc:407] Connection header received
0620 18:36:46.528194 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.528195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.528204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.528269 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.528807 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.529484 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.530514 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.530816 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.530889 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.532264 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.532266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.532266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.532272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.533848 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.534226 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.534227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.534227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.534283 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.534485 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.534486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.534487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.534606 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.534696 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.534863 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.534935 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:36:46.552577 24113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } attempt: 3
Jun 20 18:36:46 dist-test-slave-3jql krb5kdc[23957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980606, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:36:46.560338 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.552815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:33341 (local address 127.0.0.1:48242)
0620 18:36:46.552872 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.552882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.552915 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.553048 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.553049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.553212 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.553260 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.553261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.554206 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.554207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.555026 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.555029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.555499 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.555501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.555556 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.556001 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.556006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.557467 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:46.559241 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.559242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.559242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.559454 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.559646 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:46.559647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:46.559647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:46.559708 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:46.560026 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:46.560027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:46.560074 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:36:46.560144 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:36:46.560240 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:36:46.560341 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.552876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.10.109.124:33341)
0620 18:36:46.552919 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.552920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.552953 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:36:46.552973 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.552974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.552983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.553035 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.553354 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.554114 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.555113 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.555412 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.555483 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.557542 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.557543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.557544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.557550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.559150 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.559524 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.559525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.559525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.559580 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.559813 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.559814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.559815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.559949 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.560010 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.560178 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.560254 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:36:46.562234 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: Bootstrap starting.
I20260620 18:36:46.562942 24113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:46.563241 24113 log.cc:824] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:46.564046 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: No bootstrap required, opened a new log
I20260620 18:36:46.565979 24113 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.566094 24113 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:46.566124 24113 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd982374a2c44a798b699a97a2abfc9, State: Initialized, Role: FOLLOWER
I20260620 18:36:46.566241 24113 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.566552 24113 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:46.566521 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.566706 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:36:46.568329 24241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:46.568464 24241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 18:36:46.568089 24066 tablet.cc:2369] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:46.570598 24045 master_runner.cc:263] No leader master found from master bfd982374a2c44a798b699a97a2abfc9
I20260620 18:36:46.570640 24045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:36:46.582268 24196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:46.582377 24196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.582580 24196 leader_election.cc:288] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfd982374a2c44a798b699a97a2abfc9 (127.10.109.125:33775), 9a4ed520cc324fc59529f79d1cb8be3f (127.10.109.124:33341)
I20260620 18:36:46.582952 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e45df48069478fa8be0365038eec68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" is_pre_election: true
I20260620 18:36:46.583078 24162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e45df48069478fa8be0365038eec68 in term 0.
I20260620 18:36:46.583370 23988 leader_election.cc:302] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e45df48069478fa8be0365038eec68, 9a4ed520cc324fc59529f79d1cb8be3f; no voters:
I20260620 18:36:46.583850 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e45df48069478fa8be0365038eec68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd982374a2c44a798b699a97a2abfc9" is_pre_election: true
I20260620 18:36:46.583968 24087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e45df48069478fa8be0365038eec68 in term 0.
I20260620 18:36:46.584240 24196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:46.584292 24196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:46.584314 24196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:46.585134 24196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.585248 24196 leader_election.cc:288] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [CANDIDATE]: Term 1 election: Requested vote from peers bfd982374a2c44a798b699a97a2abfc9 (127.10.109.125:33775), 9a4ed520cc324fc59529f79d1cb8be3f (127.10.109.124:33341)
I20260620 18:36:46.585593 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e45df48069478fa8be0365038eec68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd982374a2c44a798b699a97a2abfc9"
I20260620 18:36:46.585657 24087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:46.586472 24087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e45df48069478fa8be0365038eec68 in term 1.
I20260620 18:36:46.586676 23989 leader_election.cc:302] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e45df48069478fa8be0365038eec68, bfd982374a2c44a798b699a97a2abfc9; no voters:
I20260620 18:36:46.586822 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e45df48069478fa8be0365038eec68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4ed520cc324fc59529f79d1cb8be3f"
I20260620 18:36:46.586895 24162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:46.587736 24162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e45df48069478fa8be0365038eec68 in term 1.
I20260620 18:36:46.588708 24196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:46.588848 24196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [term 1 LEADER]: Becoming Leader. State: Replica: 54e45df48069478fa8be0365038eec68, State: Running, Role: LEADER
I20260620 18:36:46.588923 24196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } }
I20260620 18:36:46.589253 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e45df48069478fa8be0365038eec68. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.589313 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:46.589578 24245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:46.589766 24245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:46.591646 24162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [term 1 FOLLOWER]: Refusing update from remote peer 54e45df48069478fa8be0365038eec68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:46.591849 24087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [term 1 FOLLOWER]: Refusing update from remote peer 54e45df48069478fa8be0365038eec68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:46.592259 24196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:46.592382 24196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:46.593472 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e45df48069478fa8be0365038eec68. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.593552 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:46.594462 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e45df48069478fa8be0365038eec68. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.594551 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:46.596318 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.596390 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:46.596722 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.596794 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:46.597016 24246 mvcc.cc:205] Tried to move back new op lower bound from 7298992564596764672 to 7298992564588613632. Current Snapshot: MvccSnapshot[applied={T|T < 7298992564596764672}]
I20260620 18:36:46.597374 24249 mvcc.cc:205] Tried to move back new op lower bound from 7298992564596764672 to 7298992564588613632. Current Snapshot: MvccSnapshot[applied={T|T < 7298992564596764672}]
I20260620 18:36:46.597761 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.597831 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:46.598280 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54e45df48069478fa8be0365038eec68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e45df48069478fa8be0365038eec68" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 40687 } } peers { permanent_uuid: "bfd982374a2c44a798b699a97a2abfc9" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 33775 } } peers { permanent_uuid: "9a4ed520cc324fc59529f79d1cb8be3f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 33341 } } }
I20260620 18:36:46.598351 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:46.599673 24247 mvcc.cc:205] Tried to move back new op lower bound from 7298992564596764672 to 7298992564588613632. Current Snapshot: MvccSnapshot[applied={T|T < 7298992564596764672}]
I20260620 18:36:46.600162 24245 catalog_manager.cc:1347] Generated new cluster ID: dab2892781d94bc79237cf7a6475bc8d
I20260620 18:36:46.600201 24245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:46.620002 24245 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:46.621043 24245 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:46.632179 24245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54e45df48069478fa8be0365038eec68: Generated new TSK 0
I20260620 18:36:46.632407 24245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:47.532325 24226 catalog_manager.cc:1259] Loaded cluster ID: dab2892781d94bc79237cf7a6475bc8d
I20260620 18:36:47.532390 24226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: loading cluster ID for follower catalog manager: success
I20260620 18:36:47.533201 24226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: acquiring CA information for follower catalog manager: success
I20260620 18:36:47.533493 24226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a4ed520cc324fc59529f79d1cb8be3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:36:47.568871 24241 catalog_manager.cc:1259] Loaded cluster ID: dab2892781d94bc79237cf7a6475bc8d
I20260620 18:36:47.568949 24241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: loading cluster ID for follower catalog manager: success
I20260620 18:36:47.569794 24241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: acquiring CA information for follower catalog manager: success
I20260620 18:36:47.570078 24241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfd982374a2c44a798b699a97a2abfc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:36:47.686461 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 23970
I20260620 18:36:47.691468 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24045
I20260620 18:36:47.696127 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24120
2026-06-20T18:36:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2797 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:47 dist-test-slave-3jql krb5kdc[24262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:47 dist-test-slave-3jql krb5kdc[24262](info): set up 2 sockets
Jun 20 18:36:47 dist-test-slave-3jql krb5kdc[24262](info): commencing operation
krb5kdc: starting...
W20260620 18:36:48.730829 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:36:48 dist-test-slave-3jql krb5kdc[24262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:36:48.841647 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33127
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:33127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:48.924468 24275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:48.924765 24275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:48.924817 24275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:48.926716 24275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:48.926800 24275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:48.926823 24275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:48.926856 24275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:48.926882 24275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:48.928895 24275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:33127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:48.929136 24275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:48.929353 24275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:48.934233 24281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:48.934269 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:48.934445 24275 server_base.cc:1034] running on GCE node
W20260620 18:36:48.934486 24280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:48.934916 24275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:48.935212 24275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:48.936396 24275 hybrid_clock.cc:648] HybridClock initialized: now 1781980608936380 us; error 38 us; skew 500 ppm
Jun 20 18:36:48 dist-test-slave-3jql krb5kdc[24262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:48.939792 24275 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:48.940304 24275 webserver.cc:460] Webserver started at http://127.10.109.126:43993/ using document root <none> and password file <none>
I20260620 18:36:48.940551 24275 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:48.940609 24275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:48.940747 24275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:48.942237 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "b6c8dfd318fb4efd9916df956771cb75"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-3jql"
server_key: "4ecbec79f12c2280c10c66ac319046fd"
server_key_iv: "5ce324facc90fb0205b29f3160a9b8f5"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.942711 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "b6c8dfd318fb4efd9916df956771cb75"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-3jql"
server_key: "4ecbec79f12c2280c10c66ac319046fd"
server_key_iv: "5ce324facc90fb0205b29f3160a9b8f5"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.944619 24275 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:48.945485 24290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:48.945658 24275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:48.945752 24275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "b6c8dfd318fb4efd9916df956771cb75"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-3jql"
server_key: "4ecbec79f12c2280c10c66ac319046fd"
server_key_iv: "5ce324facc90fb0205b29f3160a9b8f5"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.945834 24275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:48.970031 24275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:48.971802 24275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:48.971943 24275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:48.975715 24275 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:33127
I20260620 18:36:48.975795 24342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:33127 every 8 connection(s)
I20260620 18:36:48.976183 24275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:48.977031 24343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:48.979240 24343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75: Bootstrap starting.
I20260620 18:36:48.979769 24343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:48.980036 24343 log.cc:824] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:48.980679 24343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75: No bootstrap required, opened a new log
I20260620 18:36:48.982197 24343 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } }
I20260620 18:36:48.982354 24343 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:48.982398 24343 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c8dfd318fb4efd9916df956771cb75, State: Initialized, Role: FOLLOWER
I20260620 18:36:48.982528 24343 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } }
I20260620 18:36:48.982605 24343 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:48.982647 24343 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:48.982694 24343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:48.983192 24343 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } }
I20260620 18:36:48.983310 24343 leader_election.cc:302] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c8dfd318fb4efd9916df956771cb75; no voters:
I20260620 18:36:48.983475 24343 leader_election.cc:288] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:48.983510 24346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:48.983700 24346 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [term 1 LEADER]: Becoming Leader. State: Replica: b6c8dfd318fb4efd9916df956771cb75, State: Running, Role: LEADER
I20260620 18:36:48.983711 24343 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:48.983820 24346 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } }
I20260620 18:36:48.984169 24347 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c8dfd318fb4efd9916df956771cb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } } }
I20260620 18:36:48.984257 24347 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:48.984282 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c8dfd318fb4efd9916df956771cb75. Latest consensus state: current_term: 1 leader_uuid: "b6c8dfd318fb4efd9916df956771cb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8dfd318fb4efd9916df956771cb75" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33127 } } }
I20260620 18:36:48.984344 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:48.984484 24354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:48.985044 24354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:48.985728 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24275
I20260620 18:36:48.985823 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:36:48.986013 10677 external_mini_cluster.cc:1428] Setting key 64e1c653db0608aaeb264c861bba6cd7
I20260620 18:36:48.986574 24354 catalog_manager.cc:1347] Generated new cluster ID: 4fcb2471f2c24605b47fa489874ad43b
I20260620 18:36:48.986632 24354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:36:48 dist-test-slave-3jql krb5kdc[24262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:48.997179 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:48.986429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.10.109.126:33127)
0620 18:36:48.986578 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:48.986586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:48.986764 (+ 178us) server_negotiation.cc:407] Connection header received
0620 18:36:48.986951 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:48.986957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:48.987006 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:48.987107 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:48.987882 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.989228 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.990242 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.990872 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.990956 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.993547 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:48.993551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:48.993551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:48.993562 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:48.995460 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:48.995793 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:48.995796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:48.995796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:48.995882 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:48.996049 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:48.996049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:48.996049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:48.996312 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:48.996438 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:48.996834 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:36:48.996958 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 18:36:48.997929 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:49.002069 24354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:49.002521 24354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:49.006356 24354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c8dfd318fb4efd9916df956771cb75: Generated new TSK 0
I20260620 18:36:49.006510 24354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:49.051831 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24275
2026-06-20T18:36:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1354 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:49 dist-test-slave-3jql krb5kdc[24370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:49 dist-test-slave-3jql krb5kdc[24370](info): set up 2 sockets
Jun 20 18:36:49 dist-test-slave-3jql krb5kdc[24370](info): commencing operation
krb5kdc: starting...
W20260620 18:36:50.082871 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:36:50.195272 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:50.273955 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:50.274142 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:50.274163 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:50.275341 24383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:50.275380 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:50.275395 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:50.275429 24383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:50.275461 24383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:50.276871 24383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:35817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:50.277098 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:50.277307 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:36:50.282001 24383 server_base.cc:1034] running on GCE node
W20260620 18:36:50.281977 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.282121 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.281993 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.282359 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:50.282631 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:50.283787 24383 hybrid_clock.cc:648] HybridClock initialized: now 1781980610283776 us; error 40 us; skew 500 ppm
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:50.286161 24383 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:50.286512 24383 webserver.cc:460] Webserver started at http://127.10.109.126:37541/ using document root <none> and password file <none>
I20260620 18:36:50.286684 24383 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:50.286723 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:50.286873 24383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:50.287798 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7c297b6b8719492b887430bf653f7148"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "9738e78cb3668f360365f7e8859af47a"
server_key_iv: "3db718674493340b78c76912910b2626"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.288091 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7c297b6b8719492b887430bf653f7148"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "9738e78cb3668f360365f7e8859af47a"
server_key_iv: "3db718674493340b78c76912910b2626"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.289319 24383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:36:50.289996 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.290141 24383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:50.290220 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7c297b6b8719492b887430bf653f7148"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "9738e78cb3668f360365f7e8859af47a"
server_key_iv: "3db718674493340b78c76912910b2626"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.290273 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:50.311236 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:50.313007 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:50.313108 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:50.316792 24383 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:35817
I20260620 18:36:50.316859 24450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:35817 every 8 connection(s)
I20260620 18:36:50.317162 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:50.318147 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.319145 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24383
I20260620 18:36:50.319264 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:36:50.319454 10677 external_mini_cluster.cc:1428] Setting key bd12cda6994ca51c294fddc2afb0de50
I20260620 18:36:50.321390 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap starting.
I20260620 18:36:50.322048 24451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.322346 24451 log.cc:824] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.323062 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: No bootstrap required, opened a new log
I20260620 18:36:50.325171 24451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:50.325291 24451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.325313 24451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.325448 24451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:50.325523 24451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:50.325546 24451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:50.325609 24451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:50.326265 24451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:50.326361 24451 leader_election.cc:302] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c297b6b8719492b887430bf653f7148; no voters:
I20260620 18:36:50.326519 24451 leader_election.cc:288] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:50.326572 24456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:50.326779 24451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:50.326785 24456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 LEADER]: Becoming Leader. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Running, Role: LEADER
I20260620 18:36:50.326906 24456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:50.327179 24456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c297b6b8719492b887430bf653f7148. Latest consensus state: current_term: 1 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:50.327267 24456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:50.327195 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:50.327420 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
W20260620 18:36:50.328724 24471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:50.328778 24471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:36:50.328879 24462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:50.329093 24462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:50.330554 24462 catalog_manager.cc:1347] Generated new cluster ID: 42508eee05bc4e21b51c66ba4b8e4541
I20260620 18:36:50.330603 24462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:50.330602 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.319950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.10.109.126:35817)
0620 18:36:50.320113 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.320122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.320153 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:36:50.320682 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.320691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.320751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.320899 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.321512 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.322837 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.323556 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.324151 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.324263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.326870 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.326874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.326874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.326885 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.329061 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.329394 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.329397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.329398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.329486 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.329740 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.329741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.329741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.330026 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.330164 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.330295 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.330425 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260620 18:36:50.351083 24462 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:50.351575 24462 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:50.356575 24462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Generated new TSK 0
I20260620 18:36:50.356738 24462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:36:50.385501 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:50.464885 24477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:50.465067 24477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:50.465091 24477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:50.466208 24477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:50.466248 24477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:50.466298 24477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:36:50.467746 24477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:50.467972 24477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:50.468184 24477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:50.473207 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.473203 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.473292 24477 server_base.cc:1034] running on GCE node
W20260620 18:36:50.473204 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.473606 24477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:50.473850 24477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:50.475006 24477 hybrid_clock.cc:648] HybridClock initialized: now 1781980610474994 us; error 29 us; skew 500 ppm
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:50.477367 24477 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:36:50.477775 24477 webserver.cc:460] Webserver started at http://127.10.109.65:35581/ using document root <none> and password file <none>
I20260620 18:36:50.478014 24477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:50.478083 24477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:50.478221 24477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:50.479322 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "fb0202ba269f4a8da1a1299436d7d87e"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "e88258864e70402a9d09e4a737b892b5"
server_key_iv: "0e6be520cfec423376e6a157271ca527"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.479663 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "fb0202ba269f4a8da1a1299436d7d87e"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "e88258864e70402a9d09e4a737b892b5"
server_key_iv: "0e6be520cfec423376e6a157271ca527"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.480885 24477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:36:50.481910 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.482093 24477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:50.482188 24477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "fb0202ba269f4a8da1a1299436d7d87e"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "e88258864e70402a9d09e4a737b892b5"
server_key_iv: "0e6be520cfec423376e6a157271ca527"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.482264 24477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:50.505494 24477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:50.507335 24477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:50.507483 24477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:50.507683 24477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:50.507980 24477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:50.508033 24477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.508072 24477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:50.508101 24477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.513620 24477 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:33511
I20260620 18:36:50.513656 24605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:33511 every 8 connection(s)
I20260620 18:36:50.513983 24477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:50.519773 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24477
I20260620 18:36:50.519938 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:50.520128 10677 external_mini_cluster.cc:1428] Setting key c2a872ac645a6a00b723ce8d1d92b89f
I20260620 18:36:50.523012 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.514639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:43771 (local address 127.10.109.126:35817)
0620 18:36:50.514696 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.514698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.514925 (+ 227us) server_negotiation.cc:407] Connection header received
0620 18:36:50.515418 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.515419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.515431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.515488 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.516379 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.516880 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.517815 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.518053 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.518134 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.520217 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.520219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.520219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.520227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.521652 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.522068 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.522069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.522070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.522130 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.522375 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.522376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.522376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.522489 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.522556 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.522837 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.522921 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:36:50.523185 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.514632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:43771)
0620 18:36:50.514792 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.514832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.515314 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.515550 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.515554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.516090 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.516264 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.516268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.516969 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.516970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.517724 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.517726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.518146 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.518147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.518263 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.518659 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.518666 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.520118 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:50.521750 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.521757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.521759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.521981 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.522194 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.522195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.522196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.522301 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.522560 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:50.522567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:50.522691 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.522815 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.522973 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260620 18:36:50.523635 24606 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:50.523731 24606 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:50.523918 24606 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:50.524453 24415 ts_manager.cc:194] Registered new tserver with Master: fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511)
I20260620 18:36:50.525066 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:43771
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:36:50.544847 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:50.629168 24611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:50.629364 24611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:50.629386 24611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:50.630502 24611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:50.630543 24611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:50.630569 24611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:36:50.632032 24611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:50.632268 24611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:50.632490 24611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:50.637377 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.637403 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.637553 24611 server_base.cc:1034] running on GCE node
W20260620 18:36:50.637368 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.637820 24611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:50.638029 24611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:50.639168 24611 hybrid_clock.cc:648] HybridClock initialized: now 1781980610639154 us; error 30 us; skew 500 ppm
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:50.641535 24611 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:36:50.641988 24611 webserver.cc:460] Webserver started at http://127.10.109.66:46413/ using document root <none> and password file <none>
I20260620 18:36:50.642208 24611 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:50.642270 24611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:50.642412 24611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:50.643461 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "f25bd70242c848b38498cccb2ddd050f"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "61022a703b3796ea8a0c41141c899506"
server_key_iv: "6b171f029b169764ce36dcf94425b6c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.643810 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "f25bd70242c848b38498cccb2ddd050f"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "61022a703b3796ea8a0c41141c899506"
server_key_iv: "6b171f029b169764ce36dcf94425b6c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.645072 24611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:36:50.645789 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.646011 24611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:50.646116 24611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "f25bd70242c848b38498cccb2ddd050f"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "61022a703b3796ea8a0c41141c899506"
server_key_iv: "6b171f029b169764ce36dcf94425b6c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.646188 24611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:50.654628 24611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:50.656427 24611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:50.656580 24611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:50.656785 24611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:50.657070 24611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:50.657120 24611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.657153 24611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:50.657181 24611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.662905 24611 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:34277
I20260620 18:36:50.662958 24739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:34277 every 8 connection(s)
I20260620 18:36:50.663226 24611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:50.669509 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24611
I20260620 18:36:50.669620 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:50.669812 10677 external_mini_cluster.cc:1428] Setting key 4b28005a111dbcc0a0266b3e36a3bf2c
I20260620 18:36:50.672427 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.663934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:36455 (local address 127.10.109.126:35817)
0620 18:36:50.664024 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.664026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.664162 (+ 136us) server_negotiation.cc:407] Connection header received
0620 18:36:50.664645 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.664647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.664658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.664721 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.665628 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.666116 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.667105 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.667319 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.667398 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.669345 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.669346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.669346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.669353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.671067 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.671423 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.671424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.671424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.671483 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.671766 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.671767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.671767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.671907 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.671982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.672308 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.672366 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:36:50.672685 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.663816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:36455)
0620 18:36:50.664038 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.664080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.664548 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.664783 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.664796 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.665344 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.665525 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.665530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.666203 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.666203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.667022 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.667025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.667408 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.667409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.667521 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.667921 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.667929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.669261 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:50.671138 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.671144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.671145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.671352 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.671573 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.671574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.671574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.671699 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.671986 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:50.671993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:50.672153 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.672275 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.672444 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":126,"threads_started":1}
I20260620 18:36:50.673203 24740 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:50.673301 24740 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:50.673460 24740 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:50.673816 24415 ts_manager.cc:194] Registered new tserver with Master: f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277)
I20260620 18:36:50.674185 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:36455
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:36:50.694085 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:50.771490 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:50.771709 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:50.771746 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:50.772833 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:50.772907 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:50.772953 24745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:36:50.774336 24745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:50.774582 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:50.774811 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:50.779500 24753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.779554 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.779489 24751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.779728 24745 server_base.cc:1034] running on GCE node
I20260620 18:36:50.779979 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:50.780174 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:50.781306 24745 hybrid_clock.cc:648] HybridClock initialized: now 1781980610781294 us; error 29 us; skew 500 ppm
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:50.783613 24745 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:36:50.784005 24745 webserver.cc:460] Webserver started at http://127.10.109.67:36483/ using document root <none> and password file <none>
I20260620 18:36:50.784221 24745 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:50.784271 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:50.784356 24745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:50.785356 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "9221c3d4099d44f09bc3af270588a73a"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "f67c1d9779b6cdf81c55da0d673e293d"
server_key_iv: "37e28b49536d242218bdc8afb514e62a"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.785686 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "9221c3d4099d44f09bc3af270588a73a"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "f67c1d9779b6cdf81c55da0d673e293d"
server_key_iv: "37e28b49536d242218bdc8afb514e62a"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.786948 24745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:36:50.787637 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.787882 24745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:50.787959 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "9221c3d4099d44f09bc3af270588a73a"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "f67c1d9779b6cdf81c55da0d673e293d"
server_key_iv: "37e28b49536d242218bdc8afb514e62a"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.788008 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:50.801432 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:50.803531 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:50.803633 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:50.803787 24745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:50.804008 24745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:50.804034 24745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.804054 24745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:50.804066 24745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.809372 24745 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33041
I20260620 18:36:50.809437 24873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33041 every 8 connection(s)
I20260620 18:36:50.809713 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
Jun 20 18:36:50 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:50.818264 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24745
I20260620 18:36:50.818322 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.810359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:41971 (local address 127.10.109.126:35817)
0620 18:36:50.810420 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.810423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.810608 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:36:50.811206 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.811207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.811219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.811275 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.812137 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.812640 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.813578 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.813785 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.813889 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.815817 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.815819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.815819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.815827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.816991 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.817390 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.817392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.817392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.817452 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.817704 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.817705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.817705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.817816 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.817890 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.818158 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.818234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:36:50.818408 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:50.818506 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.810341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:41971)
0620 18:36:50.810491 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.810531 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.811111 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.811344 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.811348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.811865 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.812035 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.812039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.812729 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.812730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.813489 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.813491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.813899 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.813900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.814011 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.814376 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.814383 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.815717 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:50.817075 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.817081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.817083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.817304 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.817534 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.817535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.817535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.817631 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.817882 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:50.817888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:50.818017 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.818122 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.818268 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260620 18:36:50.818578 10677 external_mini_cluster.cc:1428] Setting key dc5637bd539ce7d2367ff0274d140317
I20260620 18:36:50.818965 24874 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:50.819061 24874 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:50.819249 24874 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:50.819566 24415 ts_manager.cc:194] Registered new tserver with Master: 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
I20260620 18:36:50.819911 24415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:41971
I20260620 18:36:50.830194 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:50.836745 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.831670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.10.109.126:35817)
0620 18:36:50.831729 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.831731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.831738 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:36:50.831756 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.831757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.831764 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.831806 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.832306 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.832753 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.833450 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.833618 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.833666 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.834688 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.834689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.834689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.834695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.835837 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.836144 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.836145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.836145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.836193 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.836354 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.836354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.836354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.836486 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.836558 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.836636 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.836688 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:36:50.838130 24415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:50.839797 24415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:50.845881 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.842146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.10.109.67:33041)
0620 18:36:50.842317 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.842322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.842330 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:36:50.842379 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.842381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.842395 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.842559 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.842730 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843767 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.844961 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845443 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845538 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845597 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.845759 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.845798 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260620 18:36:50.845875 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.841676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.0.0.1:46796)
0620 18:36:50.841783 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.841823 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.841922 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.842190 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.842194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.842197 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.842280 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.842282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843754 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.843755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.844648 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.844650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845566 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.845631 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845649 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.845693 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.845762 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":39,"thread_start_us":63,"threads_started":1}
I20260620 18:36:50.846019 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.842002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.10.109.66:34277)
0620 18:36:50.842252 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.842257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.842266 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:36:50.842305 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.842307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.842321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.842391 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.842575 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843362 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.844735 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845462 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845807 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845870 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.845895 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.845938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260620 18:36:50.846101 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.841784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.10.109.65:33511)
0620 18:36:50.842003 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.842008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.842019 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:36:50.842060 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.842062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.842078 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.842381 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.842514 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843645 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.844748 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845462 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845903 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845957 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.845979 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.846020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260620 18:36:50.845883 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.841877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:34277 (local address 127.0.0.1:55052)
0620 18:36:50.842086 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.842097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.842144 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.842386 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.842387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.842389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.842455 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.842456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843463 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.843465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.844645 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.844648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845568 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.845631 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845647 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.845693 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.845761 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":2,"thread_start_us":36,"threads_started":1}
I20260620 18:36:50.846685 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.841991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33041 (local address 127.0.0.1:59388)
0620 18:36:50.842210 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.842220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.842263 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.842608 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.842608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.842610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.842665 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.842666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.843865 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.843866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.844878 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.844879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.845541 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.845542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.845633 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.845647 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.845741 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.845785 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":46,"threads_started":1}
I20260620 18:36:50.846892 24540 tablet_service.cc:1467] Processing CreateTablet for tablet f767d942738845f2a9e7ce4afdba07a1 (DEFAULT_TABLE table=test-table [id=5ec2cdee405f4eb8999f34942226f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.847250 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767d942738845f2a9e7ce4afdba07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.847357 24674 tablet_service.cc:1467] Processing CreateTablet for tablet f767d942738845f2a9e7ce4afdba07a1 (DEFAULT_TABLE table=test-table [id=5ec2cdee405f4eb8999f34942226f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.847360 24808 tablet_service.cc:1467] Processing CreateTablet for tablet f767d942738845f2a9e7ce4afdba07a1 (DEFAULT_TABLE table=test-table [id=5ec2cdee405f4eb8999f34942226f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.847657 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767d942738845f2a9e7ce4afdba07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.847656 24808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767d942738845f2a9e7ce4afdba07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.849931 24888 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap starting.
I20260620 18:36:50.850399 24888 tablet_bootstrap.cc:654] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.850927 24890 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Bootstrap starting.
I20260620 18:36:50.850944 24888 log.cc:824] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.851541 24890 tablet_bootstrap.cc:654] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.851670 24888 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: No bootstrap required, opened a new log
I20260620 18:36:50.851747 24888 ts_tablet_manager.cc:1397] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:36:50.852067 24890 log.cc:824] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.852384 24889 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Bootstrap starting.
I20260620 18:36:50.852735 24890 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: No bootstrap required, opened a new log
I20260620 18:36:50.852813 24890 ts_tablet_manager.cc:1397] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:36:50.853018 24889 tablet_bootstrap.cc:654] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.853619 24889 log.cc:824] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.853909 24888 raft_consensus.cc:348] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.854058 24888 raft_consensus.cc:374] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.854105 24888 raft_consensus.cc:729] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0202ba269f4a8da1a1299436d7d87e, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.854223 24888 consensus_queue.cc:260] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.854478 24889 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: No bootstrap required, opened a new log
I20260620 18:36:50.854489 24888 ts_tablet_manager.cc:1428] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:36:50.854593 24606 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
I20260620 18:36:50.854681 24890 raft_consensus.cc:348] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.854547 24889 ts_tablet_manager.cc:1397] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:50.854864 24890 raft_consensus.cc:374] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.854918 24890 raft_consensus.cc:729] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9221c3d4099d44f09bc3af270588a73a, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.855042 24890 consensus_queue.cc:260] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.855316 24890 ts_tablet_manager.cc:1428] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:36:50.855324 24874 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
I20260620 18:36:50.856803 24889 raft_consensus.cc:348] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.856917 24889 raft_consensus.cc:374] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.856938 24889 raft_consensus.cc:729] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25bd70242c848b38498cccb2ddd050f, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.857034 24889 consensus_queue.cc:260] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:50.857316 24889 ts_tablet_manager.cc:1428] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:50.857326 24740 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
W20260620 18:36:50.913805 24741 tablet.cc:2369] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:51.014680 24607 tablet.cc:2369] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:51.060230 24875 tablet.cc:2369] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:51.146087 24895 raft_consensus.cc:482] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:51.146220 24895 raft_consensus.cc:504] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.146549 24895 leader_election.cc:288] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511), f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277)
I20260620 18:36:51.149657 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.146891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:59641 (local address 127.10.109.65:33511)
0620 18:36:51.146957 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.146960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.146974 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:36:51.147022 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.147023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.147034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.147086 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:51.147272 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.147857 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.148808 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.149250 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.149351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.149364 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.149561 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.149598 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:36:51.149649 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.146780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.10.109.67:59641)
0620 18:36:51.146853 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.146869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.146951 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.147097 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.147098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.147100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:51.147186 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.147195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.147935 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.147936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.148735 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.148737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.149386 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.149387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.149477 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.149481 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.149536 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.149578 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:36:51.149767 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.146945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:34277 (local address 127.10.109.67:48773)
0620 18:36:51.147108 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.147118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.147167 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.147272 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.147273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.147275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:51.147345 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.147347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.147965 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.147965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.148909 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.148911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.149476 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.149477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.149563 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.149568 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.149619 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.149654 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260620 18:36:51.149802 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.147104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:48773 (local address 127.10.109.66:34277)
0620 18:36:51.147165 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.147168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.147176 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:36:51.147206 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.147207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.147216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.147270 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:51.147405 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.147842 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.148997 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.149390 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.149462 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.149470 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.149626 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.149675 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:36:51.149991 24560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767d942738845f2a9e7ce4afdba07a1" candidate_uuid: "9221c3d4099d44f09bc3af270588a73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0202ba269f4a8da1a1299436d7d87e" is_pre_election: true
I20260620 18:36:51.150142 24560 raft_consensus.cc:2454] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9221c3d4099d44f09bc3af270588a73a in term 0.
I20260620 18:36:51.150370 24763 leader_election.cc:302] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9221c3d4099d44f09bc3af270588a73a, fb0202ba269f4a8da1a1299436d7d87e; no voters:
I20260620 18:36:51.150497 24895 raft_consensus.cc:2781] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:51.150485 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767d942738845f2a9e7ce4afdba07a1" candidate_uuid: "9221c3d4099d44f09bc3af270588a73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25bd70242c848b38498cccb2ddd050f" is_pre_election: true
I20260620 18:36:51.150563 24895 raft_consensus.cc:482] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:51.150589 24895 raft_consensus.cc:3037] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.150643 24694 raft_consensus.cc:2454] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9221c3d4099d44f09bc3af270588a73a in term 0.
I20260620 18:36:51.151376 24895 raft_consensus.cc:504] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.151499 24895 leader_election.cc:288] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [CANDIDATE]: Term 1 election: Requested vote from peers fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511), f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277)
I20260620 18:36:51.151660 24560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767d942738845f2a9e7ce4afdba07a1" candidate_uuid: "9221c3d4099d44f09bc3af270588a73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0202ba269f4a8da1a1299436d7d87e"
I20260620 18:36:51.151695 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767d942738845f2a9e7ce4afdba07a1" candidate_uuid: "9221c3d4099d44f09bc3af270588a73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25bd70242c848b38498cccb2ddd050f"
I20260620 18:36:51.151760 24694 raft_consensus.cc:3037] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.151748 24560 raft_consensus.cc:3037] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.152608 24694 raft_consensus.cc:2454] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9221c3d4099d44f09bc3af270588a73a in term 1.
I20260620 18:36:51.152602 24560 raft_consensus.cc:2454] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9221c3d4099d44f09bc3af270588a73a in term 1.
I20260620 18:36:51.152786 24763 leader_election.cc:302] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9221c3d4099d44f09bc3af270588a73a, fb0202ba269f4a8da1a1299436d7d87e; no voters:
I20260620 18:36:51.152874 24895 raft_consensus.cc:2781] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:51.153020 24895 raft_consensus.cc:686] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 LEADER]: Becoming Leader. State: Replica: 9221c3d4099d44f09bc3af270588a73a, State: Running, Role: LEADER
I20260620 18:36:51.153119 24895 consensus_queue.cc:237] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.153843 24414 catalog_manager.cc:5495] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9221c3d4099d44f09bc3af270588a73a (127.10.109.67). New cstate: current_term: 1 leader_uuid: "9221c3d4099d44f09bc3af270588a73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:51.196892 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.194431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.10.109.67:33041)
0620 18:36:51.194490 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.194493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.194502 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:36:51.194568 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.194569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.194588 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.194670 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:51.194823 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.195350 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.196092 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.196292 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.196369 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.196528 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:51.196640 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:51.196732 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.196757 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.196776 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:36:51.198666 24560 raft_consensus.cc:1261] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Refusing update from remote peer 9221c3d4099d44f09bc3af270588a73a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:51.198665 24694 raft_consensus.cc:1261] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Refusing update from remote peer 9221c3d4099d44f09bc3af270588a73a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:51.199025 24895 consensus_queue.cc:1035] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:51.199167 24898 consensus_queue.cc:1035] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:51.202092 24901 mvcc.cc:205] Tried to move back new op lower bound from 7298992583466971136 to 7298992583283712000. Current Snapshot: MvccSnapshot[applied={T|T < 7298992583466971136 or (T in {7298992583466971136})}]
I20260620 18:36:51.202184 24903 mvcc.cc:205] Tried to move back new op lower bound from 7298992583466971136 to 7298992583283712000. Current Snapshot: MvccSnapshot[applied={T|T < 7298992583466971136}]
I20260620 18:36:51.203712 24414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:51.203814 24414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:51.204559 24414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Sending DeleteTablet for 3 replicas of tablet f767d942738845f2a9e7ce4afdba07a1
I20260620 18:36:51.204852 24540 tablet_service.cc:1514] Processing DeleteTablet for tablet f767d942738845f2a9e7ce4afdba07a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:46796
I20260620 18:36:51.204873 24674 tablet_service.cc:1514] Processing DeleteTablet for tablet f767d942738845f2a9e7ce4afdba07a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55052
I20260620 18:36:51.205085 24910 tablet_replica.cc:330] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: stopping tablet replica
I20260620 18:36:51.205116 24808 tablet_service.cc:1514] Processing DeleteTablet for tablet f767d942738845f2a9e7ce4afdba07a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:59388
I20260620 18:36:51.205192 24910 raft_consensus.cc:2229] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.205309 24910 raft_consensus.cc:2258] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.205323 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24477
I20260620 18:36:51.206259 24911 tablet_replica.cc:330] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: stopping tablet replica
I20260620 18:36:51.206373 24911 raft_consensus.cc:2229] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.206475 24911 raft_consensus.cc:2258] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.206885 24911 ts_tablet_manager.cc:1905] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:51.208614 24911 ts_tablet_manager.cc:1918] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:51.208689 24911 log.cc:1196] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/f767d942738845f2a9e7ce4afdba07a1
I20260620 18:36:51.208977 24911 ts_tablet_manager.cc:1939] T f767d942738845f2a9e7ce4afdba07a1 P f25bd70242c848b38498cccb2ddd050f: Deleting consensus metadata
I20260620 18:36:51.209393 24401 catalog_manager.cc:4841] TS f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): tablet f767d942738845f2a9e7ce4afdba07a1 (table test-table [id=5ec2cdee405f4eb8999f34942226f019]) successfully deleted
W20260620 18:36:51.210165 24401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:33511 (error 108)
W20260620 18:36:51.211066 24884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:51.210518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.0.0.1:46798)
0620 18:36:51.210577 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.210589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.210631 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.211015 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 18:36:51.211202 24401 catalog_manager.cc:4584] TS fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f767d942738845f2a9e7ce4afdba07a1: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:36:51.211292 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24611
I20260620 18:36:51.216043 24912 tablet_replica.cc:330] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: stopping tablet replica
I20260620 18:36:51.216163 24912 raft_consensus.cc:2229] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:51.216264 24912 raft_consensus.cc:2258] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.216315 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24745
I20260620 18:36:51.221014 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24383
I20260620 18:36:51.221112 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.220964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33041 (local address 127.0.0.1:59414)
0620 18:36:51.221018 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.221076 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.67:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 18:36:51.226130 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--webserver_interface=127.10.109.126
--webserver_port=37541
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:51.306003 24914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.306223 24914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.306258 24914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.307407 24914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:51.307471 24914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.307497 24914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.307530 24914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:51.307554 24914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:51.308987 24914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:35817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=37541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:51.309237 24914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.309453 24914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.317946 24923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.317972 24920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.317971 24921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.318153 24914 server_base.cc:1034] running on GCE node
I20260620 18:36:51.318462 24914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.318786 24914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.319943 24914 hybrid_clock.cc:648] HybridClock initialized: now 1781980611319925 us; error 34 us; skew 500 ppm
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:51.322247 24914 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:51.322650 24914 webserver.cc:460] Webserver started at http://127.10.109.126:37541/ using document root <none> and password file <none>
I20260620 18:36:51.322909 24914 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.322966 24914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.324226 24914 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:51.324775 24930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.324977 24914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:51.325063 24914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7c297b6b8719492b887430bf653f7148"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "9738e78cb3668f360365f7e8859af47a"
server_key_iv: "3db718674493340b78c76912910b2626"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.325423 24914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:51.338394 24914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:51.340320 24914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:51.340440 24914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:51.344357 24914 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:35817
I20260620 18:36:51.344405 24982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:35817 every 8 connection(s)
I20260620 18:36:51.344754 24914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:51.346302 24983 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:51.346998 24983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap starting.
I20260620 18:36:51.350596 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24914
I20260620 18:36:51.350921 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:33511
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=35581
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:51.355436 24983 log.cc:824] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:51.357165 24983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:51.357447 24983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap complete.
I20260620 18:36:51.361127 24983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:51.361493 24983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.361691 24983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:51.361774 24983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:51.361829 24983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:51.361882 24983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:51.363265 24983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:51.363382 24983 leader_election.cc:302] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c297b6b8719492b887430bf653f7148; no voters:
I20260620 18:36:51.363536 24983 leader_election.cc:288] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:36:51.363698 24988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:36:51.363776 24983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:51.363903 24988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 LEADER]: Becoming Leader. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Running, Role: LEADER
I20260620 18:36:51.364046 24988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:51.364331 24988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:51.364521 24988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:51.364681 24988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c297b6b8719492b887430bf653f7148. Latest consensus state: current_term: 2 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:51.364768 24988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:51.364892 24996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:51.365634 24996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:51.365731 24996 catalog_manager.cc:1259] Loaded cluster ID: 42508eee05bc4e21b51c66ba4b8e4541
I20260620 18:36:51.365761 24996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:51.365983 24996 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:36:51.371043 24996 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:36:51.373402 24996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:51.373589 24996 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Loaded TSK: 0
I20260620 18:36:51.373790 24996 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:36:51.438241 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.438429 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.438452 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.439635 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.439675 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.439702 24985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:36:51.441133 24985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:33511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=35581
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:51.441360 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.441579 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.447194 25011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.447249 25013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.447397 25010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.447616 24985 server_base.cc:1034] running on GCE node
I20260620 18:36:51.447808 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.448048 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.449204 24985 hybrid_clock.cc:648] HybridClock initialized: now 1781980611449192 us; error 29 us; skew 500 ppm
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:51.451835 24985 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:36:51.452315 24985 webserver.cc:460] Webserver started at http://127.10.109.65:35581/ using document root <none> and password file <none>
I20260620 18:36:51.452569 24985 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.452646 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.454337 24985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:51.455083 25020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.455307 24985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:51.455399 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "fb0202ba269f4a8da1a1299436d7d87e"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "e88258864e70402a9d09e4a737b892b5"
server_key_iv: "0e6be520cfec423376e6a157271ca527"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.455799 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:51.467178 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:51.469487 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:51.469631 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:51.469861 24985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:51.470309 25027 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:36:51.471843 24985 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:36:51.471901 24985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:36:51.471932 24985 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:36:51.472708 24985 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:36:51.472754 24985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:51.472810 25027 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap starting.
I20260620 18:36:51.479741 24985 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:33511
I20260620 18:36:51.479908 25134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:33511 every 8 connection(s)
I20260620 18:36:51.480170 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:51.483318 25027 log.cc:824] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:51.485908 25027 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:51.485960 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 24985
I20260620 18:36:51.486248 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:34277
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=46413
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:51.486281 25027 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap complete.
I20260620 18:36:51.487690 25027 ts_tablet_manager.cc:1397] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:51.489754 25027 raft_consensus.cc:348] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.490056 25027 raft_consensus.cc:729] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0202ba269f4a8da1a1299436d7d87e, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.490190 25027 consensus_queue.cc:260] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.490414 25027 ts_tablet_manager.cc:1428] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:36:51.494552 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.480931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:47809)
0620 18:36:51.481132 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.481181 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.482157 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.482651 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.482657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.483101 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.483368 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.483375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.485067 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.485068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.485956 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.485960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.486971 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.486972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.487110 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.487698 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.487708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.489561 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:51.492631 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.492637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.492638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.492933 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.493228 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.493229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.493229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.493356 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.493826 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:51.493833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:51.493986 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.494112 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.494299 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260620 18:36:51.494562 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.481260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:47809 (local address 127.10.109.126:35817)
0620 18:36:51.481453 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.481461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.481503 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:36:51.482386 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.482393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.482446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.482585 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.483510 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.484937 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.486066 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.486698 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.486860 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.489691 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.489695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.489696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.489708 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.492537 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.493022 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.493025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.493026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.493145 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.493428 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.493429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.493430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.493735 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.493832 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.494185 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.494328 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260620 18:36:51.495523 25135 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:51.495627 25135 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:51.495811 25135 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:51.496567 24947 ts_manager.cc:194] Registered new tserver with Master: fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511)
I20260620 18:36:51.497676 24947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:47809
I20260620 18:36:51.498520 25135 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:36:51.506969 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.497385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.10.109.65:33511)
0620 18:36:51.497854 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.497858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.497867 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:36:51.497981 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.497984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.498008 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.498705 (+ 697us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.499284 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.500374 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.501523 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.501918 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.501988 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.504280 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.504286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.504286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.504292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.505709 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.506132 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.506134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.506134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.506185 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.506386 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.506386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.506386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.506530 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.506590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.506844 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.506905 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":76,"threads_started":1}
I20260620 18:36:51.506969 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.497151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.0.0.1:46814)
0620 18:36:51.497550 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.497589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.497656 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.498784 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.498790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.499159 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.499208 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.499211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.500459 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.500460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.501433 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.501435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.501995 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.501996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.502070 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.502500 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.502505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.504185 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:51.505785 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.505791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.505793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.506050 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.506251 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.506252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.506253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.506322 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.506593 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:51.506599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:51.506648 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.506731 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.506865 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":68,"threads_started":1}
I20260620 18:36:51.507695 25051 tablet_service.cc:1514] Processing DeleteTablet for tablet f767d942738845f2a9e7ce4afdba07a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:46814
I20260620 18:36:51.508220 25144 tablet_replica.cc:330] stopping tablet replica
I20260620 18:36:51.508332 25144 raft_consensus.cc:2229] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.508394 25144 raft_consensus.cc:2258] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.508661 25144 ts_tablet_manager.cc:1905] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:51.510630 25144 ts_tablet_manager.cc:1918] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:51.510689 25144 log.cc:1196] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/f767d942738845f2a9e7ce4afdba07a1
I20260620 18:36:51.510929 25144 ts_tablet_manager.cc:1939] T f767d942738845f2a9e7ce4afdba07a1 P fb0202ba269f4a8da1a1299436d7d87e: Deleting consensus metadata
I20260620 18:36:51.511415 24933 catalog_manager.cc:4841] TS fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511): tablet f767d942738845f2a9e7ce4afdba07a1 (table test-table [id=5ec2cdee405f4eb8999f34942226f019]) successfully deleted
W20260620 18:36:51.570971 25140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.571218 25140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.571257 25140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.572363 25140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.572434 25140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.572479 25140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:36:51.573870 25140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:34277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=46413
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:51.574110 25140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.574326 25140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.579116 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.579130 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.579111 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.579350 25140 server_base.cc:1034] running on GCE node
I20260620 18:36:51.579582 25140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.579819 25140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.580971 25140 hybrid_clock.cc:648] HybridClock initialized: now 1781980611580970 us; error 44 us; skew 500 ppm
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:51.583505 25140 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:36:51.583894 25140 webserver.cc:460] Webserver started at http://127.10.109.66:46413/ using document root <none> and password file <none>
I20260620 18:36:51.584103 25140 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.584177 25140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.585479 25140 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:51.586058 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.586229 25140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.586324 25140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "f25bd70242c848b38498cccb2ddd050f"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "61022a703b3796ea8a0c41141c899506"
server_key_iv: "6b171f029b169764ce36dcf94425b6c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.586622 25140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:51.597205 25140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:51.598901 25140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:51.599037 25140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:51.599244 25140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:51.599524 25140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:51.599573 25140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.599612 25140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:51.599638 25140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.605607 25140 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:34277
I20260620 18:36:51.605652 25272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:34277 every 8 connection(s)
I20260620 18:36:51.605957 25140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:51.611582 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25140
I20260620 18:36:51.611824 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:33041
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=36483
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:51.615309 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.606613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:58433 (local address 127.10.109.126:35817)
0620 18:36:51.606677 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.606680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.606927 (+ 247us) server_negotiation.cc:407] Connection header received
0620 18:36:51.607354 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.607355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.607367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.607427 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.608274 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.608807 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.609906 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.610109 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.610172 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.612414 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.612416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.612416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.612425 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.613927 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.614328 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.614330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.614330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.614391 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.614639 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.614640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.614640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.614799 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.614861 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.615122 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.615217 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:36:51.615473 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.606615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:58433)
0620 18:36:51.606805 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.606838 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.607266 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.607475 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.607478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.608012 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.608185 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.608189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.608897 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.608898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.609754 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.609757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.610185 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.610186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.610305 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.610787 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.610796 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.612313 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:51.614020 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.614026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.614028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.614239 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.614464 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.614465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.614465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.614568 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.614863 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:51.614870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:51.614997 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.615099 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.615245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260620 18:36:51.616151 25273 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:51.616243 25273 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:51.616413 25273 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:51.616797 24947 ts_manager.cc:194] Registered new tserver with Master: f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277)
I20260620 18:36:51.617172 24947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:58433
W20260620 18:36:51.694003 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.694224 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.694264 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.695401 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.695462 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.695508 25276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:36:51.697082 25276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:33041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=36483
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:51.697328 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.697537 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.702345 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.702368 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.702544 25276 server_base.cc:1034] running on GCE node
W20260620 18:36:51.702389 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.702860 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.703105 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.704213 25276 hybrid_clock.cc:648] HybridClock initialized: now 1781980611704198 us; error 26 us; skew 500 ppm
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:51.706645 25276 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:36:51.707063 25276 webserver.cc:460] Webserver started at http://127.10.109.67:36483/ using document root <none> and password file <none>
I20260620 18:36:51.707276 25276 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.707372 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.708642 25276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:51.709211 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.709373 25276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:36:51.709465 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "9221c3d4099d44f09bc3af270588a73a"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "f67c1d9779b6cdf81c55da0d673e293d"
server_key_iv: "37e28b49536d242218bdc8afb514e62a"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.709766 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:51.728597 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:51.730463 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:51.730604 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:51.730844 25276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:51.731293 25298 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:36:51.732550 25276 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:36:51.732606 25276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:51.732649 25276 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:36:51.733213 25276 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:36:51.733265 25276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:51.733433 25298 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Bootstrap starting.
I20260620 18:36:51.739771 25276 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33041
I20260620 18:36:51.739861 25405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33041 every 8 connection(s)
I20260620 18:36:51.740164 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:51.743814 25298 log.cc:824] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:51.746227 25298 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:51.746527 25298 tablet_bootstrap.cc:492] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Bootstrap complete.
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:36:51.747155 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25276
I20260620 18:36:51.747632 25298 ts_tablet_manager.cc:1397] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.011s
I20260620 18:36:51.748886 25298 raft_consensus.cc:348] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.749117 25298 raft_consensus.cc:729] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9221c3d4099d44f09bc3af270588a73a, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.749270 25298 consensus_queue.cc:260] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } } peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } }
I20260620 18:36:51.749528 25298 ts_tablet_manager.cc:1428] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:36:51.750598 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.740913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:51491 (local address 127.10.109.126:35817)
0620 18:36:51.740968 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.740970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.741187 (+ 217us) server_negotiation.cc:407] Connection header received
0620 18:36:51.741961 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.741962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.741972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.742015 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.742983 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.743463 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.744848 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.745031 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.745089 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.747572 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.747574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.747574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.747581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.749139 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.749605 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.749606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.749607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.749664 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.749910 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.749911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.749911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.750024 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.750097 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.750399 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.750471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 18:36:51.750727 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.740842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:51491)
0620 18:36:51.741047 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.741091 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.741859 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.742088 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.742093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.742561 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.742873 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.742882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.743573 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.743574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.744759 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.744762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.745125 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.745126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.745252 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.745731 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.745740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.747459 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:51.749230 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.749237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.749239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.749535 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.749734 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.749735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.749735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.749844 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.750089 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:51.750095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:51.750237 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.750351 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.750508 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260620 18:36:51.751260 25406 heartbeater.cc:347] Connected to a master server at 127.10.109.126:35817
I20260620 18:36:51.751366 25406 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:51.751555 25406 heartbeater.cc:511] Master 127.10.109.126:35817 requested a full tablet report, sending...
I20260620 18:36:51.752164 24947 ts_manager.cc:194] Registered new tserver with Master: 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
I20260620 18:36:51.752758 24947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:51491
I20260620 18:36:51.753525 25406 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
I20260620 18:36:51.755923 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.747503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.10.109.126:35817)
0620 18:36:51.747604 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.747606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.747739 (+ 133us) server_negotiation.cc:407] Connection header received
0620 18:36:51.747843 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.747844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.747862 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.747898 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.748618 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.749376 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.750169 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.750461 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.750528 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.752152 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.752154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.752154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.752161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.754670 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.755034 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.755035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.755035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.755097 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.755316 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.755317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.755318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.755503 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.755647 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.755727 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.755832 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.67@KRBTEST.COM
I20260620 18:36:51.756731 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:51.759727 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.757064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.10.109.126:35817)
0620 18:36:51.757102 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.757103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.757170 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:36:51.757219 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.757219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.757256 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.757345 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:51.757432 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.758101 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.758860 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.759158 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.759211 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.759391 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:51.759509 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:51.759596 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.759644 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.759663 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:36:51.760102 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.752448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33041 (local address 127.0.0.1:59426)
0620 18:36:51.752488 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.752500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.752551 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.752958 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.752959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.753171 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.753207 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.753210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.754217 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.754218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.754691 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.754693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.755251 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.755252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.755326 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.755718 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.755723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.756980 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:51.758787 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.758788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.758789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.758991 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.759318 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:51.759318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:51.759318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:51.759401 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:51.759730 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:51.759731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:51.759775 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.759850 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.759946 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 18:36:51.760241 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.752644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.10.109.67:33041)
0620 18:36:51.752791 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.752796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.752806 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:36:51.752838 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.752845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.752858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.752939 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.753278 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.754148 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.754824 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.755181 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.755242 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.757134 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.757140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.757141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:51.757165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.758670 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.759146 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.759148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.759149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.759240 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:51.759466 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:51.759467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:51.759467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:51.759667 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.759747 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.760042 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.760130 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 18:36:51.760648 25340 tablet_service.cc:1514] Processing DeleteTablet for tablet f767d942738845f2a9e7ce4afdba07a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:59426
I20260620 18:36:51.761057 25415 tablet_replica.cc:330] stopping tablet replica
I20260620 18:36:51.761168 25415 raft_consensus.cc:2229] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.761229 25415 raft_consensus.cc:2258] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.761488 25415 ts_tablet_manager.cc:1905] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:51.759969 24947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:51.761973 24947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:51.763645 25415 ts_tablet_manager.cc:1918] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:51.763917 25415 log.cc:1196] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/f767d942738845f2a9e7ce4afdba07a1
I20260620 18:36:51.764029 25051 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 (DEFAULT_TABLE table=test-table [id=895d007ebe0a49dfac674f5cea435dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:51.764196 25051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea7212449c41e5a2b1dbeda8def2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:51.764210 25415 ts_tablet_manager.cc:1939] T f767d942738845f2a9e7ce4afdba07a1 P 9221c3d4099d44f09bc3af270588a73a: Deleting consensus metadata
I20260620 18:36:51.764623 24933 catalog_manager.cc:4841] TS 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): tablet f767d942738845f2a9e7ce4afdba07a1 (table test-table [id=5ec2cdee405f4eb8999f34942226f019]) successfully deleted
I20260620 18:36:51.765879 25027 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap starting.
I20260620 18:36:51.766028 25340 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 (DEFAULT_TABLE table=test-table [id=895d007ebe0a49dfac674f5cea435dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:51.766152 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea7212449c41e5a2b1dbeda8def2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:51.766309 25027 tablet_bootstrap.cc:654] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:51.767220 25027 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: No bootstrap required, opened a new log
I20260620 18:36:51.767272 25027 ts_tablet_manager.cc:1397] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:51.767406 25027 raft_consensus.cc:348] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.767472 25027 raft_consensus.cc:374] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:51.767526 25027 raft_consensus.cc:729] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0202ba269f4a8da1a1299436d7d87e, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.767575 25298 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Bootstrap starting.
I20260620 18:36:51.767586 25027 consensus_queue.cc:260] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.767690 25027 ts_tablet_manager.cc:1428] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:51.767863 25298 tablet_bootstrap.cc:654] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:51.768546 25298 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: No bootstrap required, opened a new log
I20260620 18:36:51.768596 25298 ts_tablet_manager.cc:1397] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:51.768750 25298 raft_consensus.cc:348] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.768821 25298 raft_consensus.cc:374] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:51.768882 25298 raft_consensus.cc:729] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9221c3d4099d44f09bc3af270588a73a, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.768930 25298 consensus_queue.cc:260] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.769024 25298 ts_tablet_manager.cc:1428] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.769457 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.763691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:34277 (local address 127.0.0.1:55056)
0620 18:36:51.763741 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.763753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.763796 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.764919 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.764919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.764921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:51.764965 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.764966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.766199 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.766200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.767460 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.767463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.769177 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.769178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.769244 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.769268 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.769356 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.769392 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:36:51.769506 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.763784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.10.109.66:34277)
0620 18:36:51.764719 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.764723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.764733 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:36:51.764770 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.764772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.764787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.764908 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:51.765029 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.766097 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.768353 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.769108 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.769180 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.769237 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.769371 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.769414 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":85,"threads_started":1}
I20260620 18:36:51.770095 25207 tablet_service.cc:1467] Processing CreateTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 (DEFAULT_TABLE table=test-table [id=895d007ebe0a49dfac674f5cea435dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:51.770339 25207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ea7212449c41e5a2b1dbeda8def2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:51.772915 25419 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Bootstrap starting.
I20260620 18:36:51.773481 25419 tablet_bootstrap.cc:654] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:51.773895 25419 log.cc:824] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:51.774655 25419 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: No bootstrap required, opened a new log
I20260620 18:36:51.774715 25419 ts_tablet_manager.cc:1397] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:36:51.775975 25419 raft_consensus.cc:348] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.776093 25419 raft_consensus.cc:374] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:51.776113 25419 raft_consensus.cc:729] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25bd70242c848b38498cccb2ddd050f, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.776219 25419 consensus_queue.cc:260] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.776432 25419 ts_tablet_manager.cc:1428] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:51.776494 25273 heartbeater.cc:503] Master 127.10.109.126:35817 was elected leader, sending a full tablet report...
I20260620 18:36:51.842912 25421 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:51.843070 25421 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.843427 25421 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041), fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511)
I20260620 18:36:51.847469 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.843644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33041 (local address 127.10.109.66:34503)
0620 18:36:51.843928 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.843939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.844027 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.844207 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.844208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.844210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:51.844282 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.844284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.845272 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.845273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.846533 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.846535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.847255 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.847256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.847325 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.847329 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.847380 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.847410 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260620 18:36:51.847510 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.843730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:34503 (local address 127.10.109.67:33041)
0620 18:36:51.843772 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.843774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.843996 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:36:51.844097 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.844098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.844136 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.844192 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:51.844377 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.845188 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.846615 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.847179 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.847263 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.847334 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.847387 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.847427 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:36:51.847796 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea7212449c41e5a2b1dbeda8def2f9" candidate_uuid: "f25bd70242c848b38498cccb2ddd050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9221c3d4099d44f09bc3af270588a73a" is_pre_election: true
I20260620 18:36:51.847958 25360 raft_consensus.cc:2454] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f25bd70242c848b38498cccb2ddd050f in term 0.
I20260620 18:36:51.848230 25162 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f; no voters:
I20260620 18:36:51.848381 25421 raft_consensus.cc:2781] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:51.848455 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.843958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:41487 (local address 127.10.109.65:33511)
0620 18:36:51.844019 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.844021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.845129 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 18:36:51.845190 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.845191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.845212 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.845284 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:51.845468 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.846185 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.847295 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.847912 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.848062 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.848133 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.848330 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.848389 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:36:51.848464 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.843758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.10.109.66:41487)
0620 18:36:51.845064 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.845076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.845128 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.845328 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.845329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.845331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:51.845380 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.845382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.846274 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.846274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.847189 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.847191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.848169 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.848170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.848246 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.848249 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:36:51.848320 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:36:51.848366 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":94,"threads_started":1}
I20260620 18:36:51.848608 25421 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:51.848713 25421 raft_consensus.cc:3037] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.848868 25089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea7212449c41e5a2b1dbeda8def2f9" candidate_uuid: "f25bd70242c848b38498cccb2ddd050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0202ba269f4a8da1a1299436d7d87e" is_pre_election: true
I20260620 18:36:51.848991 25089 raft_consensus.cc:2454] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f25bd70242c848b38498cccb2ddd050f in term 0.
I20260620 18:36:51.849535 25421 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.849665 25421 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [CANDIDATE]: Term 1 election: Requested vote from peers 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041), fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511)
I20260620 18:36:51.849834 25089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea7212449c41e5a2b1dbeda8def2f9" candidate_uuid: "f25bd70242c848b38498cccb2ddd050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0202ba269f4a8da1a1299436d7d87e"
I20260620 18:36:51.849834 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ea7212449c41e5a2b1dbeda8def2f9" candidate_uuid: "f25bd70242c848b38498cccb2ddd050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9221c3d4099d44f09bc3af270588a73a"
I20260620 18:36:51.849916 25360 raft_consensus.cc:3037] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.849916 25089 raft_consensus.cc:3037] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.850759 25360 raft_consensus.cc:2454] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f25bd70242c848b38498cccb2ddd050f in term 1.
I20260620 18:36:51.850787 25089 raft_consensus.cc:2454] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f25bd70242c848b38498cccb2ddd050f in term 1.
I20260620 18:36:51.850917 25162 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f; no voters:
I20260620 18:36:51.851020 25421 raft_consensus.cc:2781] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:51.851155 25421 raft_consensus.cc:686] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 1 LEADER]: Becoming Leader. State: Replica: f25bd70242c848b38498cccb2ddd050f, State: Running, Role: LEADER
I20260620 18:36:51.851251 25421 consensus_queue.cc:237] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:51.851990 24947 catalog_manager.cc:5495] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f reported cstate change: term changed from 0 to 1, leader changed from <none> to f25bd70242c848b38498cccb2ddd050f (127.10.109.66). New cstate: current_term: 1 leader_uuid: "f25bd70242c848b38498cccb2ddd050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } health_report { overall_health: UNKNOWN } } }
W20260620 18:36:51.856416 25274 tablet.cc:2369] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:51.861344 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.859134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.10.109.66:34277)
0620 18:36:51.859186 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.859188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.859224 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:36:51.859292 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.859292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.859318 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.859380 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:51.859532 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.859968 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.860711 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.860897 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.860952 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.861066 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:51.861145 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:51.861197 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.861226 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.861241 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:36:51.863085 25089 raft_consensus.cc:1261] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Refusing update from remote peer f25bd70242c848b38498cccb2ddd050f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:51.863125 25360 raft_consensus.cc:1261] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Refusing update from remote peer f25bd70242c848b38498cccb2ddd050f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:51.863463 25421 consensus_queue.cc:1035] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:51.863587 25423 consensus_queue.cc:1035] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:51.866248 25427 mvcc.cc:205] Tried to move back new op lower bound from 7298992586188439552 to 7298992586143420416. Current Snapshot: MvccSnapshot[applied={T|T < 7298992586188439552}]
I20260620 18:36:51.866430 25426 mvcc.cc:205] Tried to move back new op lower bound from 7298992586188439552 to 7298992586143420416. Current Snapshot: MvccSnapshot[applied={T|T < 7298992586188439552 or (T in {7298992586188439552})}]
I20260620 18:36:51.866727 25428 mvcc.cc:205] Tried to move back new op lower bound from 7298992586188439552 to 7298992586143420416. Current Snapshot: MvccSnapshot[applied={T|T < 7298992586188439552 or (T in {7298992586188439552})}]
I20260620 18:36:51.868240 24947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:51.868326 24947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:51.869001 24947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Sending DeleteTablet for 3 replicas of tablet 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:51.869293 25340 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:59426
I20260620 18:36:51.869355 25207 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55056
I20260620 18:36:51.869395 25415 tablet_replica.cc:330] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: stopping tablet replica
I20260620 18:36:51.869441 25415 raft_consensus.cc:2229] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.869505 25415 raft_consensus.cc:2258] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.869525 25051 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ea7212449c41e5a2b1dbeda8def2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:51 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:46814
I20260620 18:36:51.869612 25144 tablet_replica.cc:330] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: stopping tablet replica
I20260620 18:36:51.869661 25144 raft_consensus.cc:2229] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:51.869719 25144 raft_consensus.cc:2258] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.869829 25415 ts_tablet_manager.cc:1905] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:51.869836 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24985
I20260620 18:36:51.870867 25435 tablet_replica.cc:330] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: stopping tablet replica
I20260620 18:36:51.870971 25435 raft_consensus.cc:2229] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:51.871073 25435 raft_consensus.cc:2258] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:51.871466 25415 ts_tablet_manager.cc:1918] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:51.871503 25415 log.cc:1196] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:51.871652 25415 ts_tablet_manager.cc:1939] T 79ea7212449c41e5a2b1dbeda8def2f9 P 9221c3d4099d44f09bc3af270588a73a: Deleting consensus metadata
I20260620 18:36:51.872169 25435 ts_tablet_manager.cc:1905] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:51.872431 24933 catalog_manager.cc:4841] TS 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): tablet 79ea7212449c41e5a2b1dbeda8def2f9 (table test-table [id=895d007ebe0a49dfac674f5cea435dee]) successfully deleted
I20260620 18:36:51.873975 25435 ts_tablet_manager.cc:1918] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:51.874048 25435 log.cc:1196] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:51.874305 25435 ts_tablet_manager.cc:1939] T 79ea7212449c41e5a2b1dbeda8def2f9 P f25bd70242c848b38498cccb2ddd050f: Deleting consensus metadata
I20260620 18:36:51.874666 24933 catalog_manager.cc:4841] TS f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): tablet 79ea7212449c41e5a2b1dbeda8def2f9 (table test-table [id=895d007ebe0a49dfac674f5cea435dee]) successfully deleted
W20260620 18:36:51.876260 24933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:33511 (error 108)
I20260620 18:36:51.876636 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 25140
I20260620 18:36:51.876785 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.876647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:33511 (local address 127.0.0.1:46816)
0620 18:36:51.876708 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.876750 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 18:36:51.877908 24933 catalog_manager.cc:4584] TS fb0202ba269f4a8da1a1299436d7d87e (127.10.109.65:33511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79ea7212449c41e5a2b1dbeda8def2f9: Network error: Client connection negotiation failed: client connection to 127.10.109.65:33511: connect: Connection refused (error 111)
I20260620 18:36:51.881085 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 25276
I20260620 18:36:51.885774 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 24914
I20260620 18:36:51.891170 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--webserver_interface=127.10.109.126
--webserver_port=37541
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:51.972517 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.972765 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.972803 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.973946 25437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:51.974012 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.974036 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.974063 25437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:51.974093 25437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:51.975819 25437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:35817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=37541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:51.976125 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.976365 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.981495 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.981560 25437 server_base.cc:1034] running on GCE node
W20260620 18:36:51.981525 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.981527 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.981958 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.982184 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.983335 25437 hybrid_clock.cc:648] HybridClock initialized: now 1781980611983309 us; error 38 us; skew 500 ppm
Jun 20 18:36:51 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:51.985818 25437 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:36:51.986251 25437 webserver.cc:460] Webserver started at http://127.10.109.126:37541/ using document root <none> and password file <none>
I20260620 18:36:51.986518 25437 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.986596 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.988106 25437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:51.988895 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.989130 25437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:51.989229 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7c297b6b8719492b887430bf653f7148"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "9738e78cb3668f360365f7e8859af47a"
server_key_iv: "3db718674493340b78c76912910b2626"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.989610 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.027812 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.030359 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.030536 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.034878 25437 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:35817
I20260620 18:36:52.034927 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:35817 every 8 connection(s)
I20260620 18:36:52.035329 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:52.035799 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25437
I20260620 18:36:52.036094 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:33511
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=35581
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:52.037048 25505 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:52.037705 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap starting.
I20260620 18:36:52.046343 25505 log.cc:824] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:52.048688 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:52.049021 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148: Bootstrap complete.
I20260620 18:36:52.051370 25505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:52.051620 25505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Initialized, Role: FOLLOWER
I20260620 18:36:52.051826 25505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:52.051932 25505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:52.051986 25505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:52.052034 25505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:36:52.053001 25505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:52.053121 25505 leader_election.cc:302] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c297b6b8719492b887430bf653f7148; no voters:
I20260620 18:36:52.053325 25505 leader_election.cc:288] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:36:52.053359 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:36:52.053484 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [term 3 LEADER]: Becoming Leader. State: Replica: 7c297b6b8719492b887430bf653f7148, State: Running, Role: LEADER
I20260620 18:36:52.053593 25505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:52.053598 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } }
I20260620 18:36:52.053895 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c297b6b8719492b887430bf653f7148. Latest consensus state: current_term: 3 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:52.053953 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:52.054112 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7c297b6b8719492b887430bf653f7148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c297b6b8719492b887430bf653f7148" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 35817 } } }
I20260620 18:36:52.054176 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c297b6b8719492b887430bf653f7148 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:52.054441 25513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:52.055308 25513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:52.055430 25513 catalog_manager.cc:1259] Loaded cluster ID: 42508eee05bc4e21b51c66ba4b8e4541
I20260620 18:36:52.055511 25513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:36:52.055898 25513 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdd943d39ad google::LogMessage::Fail() at ??:0
@ 0x7fdd943d688a google::LogMessage::SendToLog() at ??:0
@ 0x7fdd943d34de google::LogMessage::Flush() at ??:0
@ 0x7fdd943d4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdd9c2f40f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdd9c2fd8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdd962a10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdd96296954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdd970fa6db start_thread at ??:0
@ 0x7fdd92ed161f clone at ??:0
W20260620 18:36:52.129803 25507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.130024 25507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.130064 25507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.131354 25507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.131408 25507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.131453 25507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:36:52.133051 25507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:33511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=35581
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:52.133302 25507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.133535 25507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.138301 25544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.138309 25541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.138391 25542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.138700 25507 server_base.cc:1034] running on GCE node
I20260620 18:36:52.138924 25507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.139194 25507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.140383 25507 hybrid_clock.cc:648] HybridClock initialized: now 1781980612140365 us; error 32 us; skew 500 ppm
Jun 20 18:36:52 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980612, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:52.143442 25507 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:36:52.143962 25507 webserver.cc:460] Webserver started at http://127.10.109.65:35581/ using document root <none> and password file <none>
I20260620 18:36:52.144239 25507 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.144308 25507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.145998 25507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:52.146878 25551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.147058 25507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:52.147190 25507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "fb0202ba269f4a8da1a1299436d7d87e"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "e88258864e70402a9d09e4a737b892b5"
server_key_iv: "0e6be520cfec423376e6a157271ca527"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.147507 25507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.165273 25507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.168031 25507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.168231 25507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.168491 25507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.168982 25558 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:36:52.170874 25507 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:36:52.170933 25507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:36:52.170971 25507 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:36:52.171797 25507 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:36:52.171844 25507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:52.171909 25558 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap starting.
I20260620 18:36:52.179036 25507 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:33511
I20260620 18:36:52.179212 25665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:33511 every 8 connection(s)
I20260620 18:36:52.179457 25507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:52.181610 25558 log.cc:824] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:52.181900 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25507
I20260620 18:36:52.182189 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:34277
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=46413
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:52.184079 25558 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:52.184396 25558 tablet_bootstrap.cc:492] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Bootstrap complete.
I20260620 18:36:52.185349 25558 ts_tablet_manager.cc:1397] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260620 18:36:52.186322 25558 raft_consensus.cc:348] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:52.186549 25558 raft_consensus.cc:729] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0202ba269f4a8da1a1299436d7d87e, State: Initialized, Role: FOLLOWER
I20260620 18:36:52.186687 25558 consensus_queue.cc:260] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:52.187017 25558 ts_tablet_manager.cc:1428] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 18:36:52.245847 25668 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:52.180124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:55747)
0620 18:36:52.180345 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.180390 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.180971 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.245632 (+ 64661us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
W20260620 18:36:52.246120 25666 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:35817 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:35817: Client connection negotiation failed: client connection to 127.10.109.126:35817: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:36:52.246445 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.246325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:48459)
0620 18:36:52.246382 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.246425 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:36:52.246845 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.246733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:33373)
0620 18:36:52.246794 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.246822 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:36:52.246982 25666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:52.271911 25670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.272119 25670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.272158 25670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.273262 25670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.273324 25670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.273373 25670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:36:52.274830 25670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:34277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=46413
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:52.275069 25670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.275281 25670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.280089 25679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.280112 25677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.280128 25676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.280305 25670 server_base.cc:1034] running on GCE node
I20260620 18:36:52.280506 25670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.280740 25670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.281912 25670 hybrid_clock.cc:648] HybridClock initialized: now 1781980612281896 us; error 27 us; skew 500 ppm
Jun 20 18:36:52 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980612, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:52.284480 25670 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:36:52.284832 25670 webserver.cc:460] Webserver started at http://127.10.109.66:46413/ using document root <none> and password file <none>
I20260620 18:36:52.285003 25670 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.285043 25670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.286238 25670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:52.286847 25686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.287020 25670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:36:52.287115 25670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "f25bd70242c848b38498cccb2ddd050f"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "61022a703b3796ea8a0c41141c899506"
server_key_iv: "6b171f029b169764ce36dcf94425b6c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.287424 25670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.299237 25670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.301309 25670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.301455 25670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.301653 25670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.301931 25670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:52.301981 25670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.302019 25670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:52.302047 25670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.307674 25670 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:34277
I20260620 18:36:52.307739 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:34277 every 8 connection(s)
I20260620 18:36:52.308071 25670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:52.308699 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25670
I20260620 18:36:52.308974 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:33041
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:35817
--webserver_port=36483
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:52.309209 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.308780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:56317)
0620 18:36:52.308928 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.308996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
W20260620 18:36:52.309561 25800 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:35817 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:35817: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
I20260620 18:36:52.309850 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.309757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:53093)
0620 18:36:52.309803 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.309832 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:36:52.310112 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.310061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:43203)
0620 18:36:52.310083 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.310100 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 18:36:52.310218 25800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:52.390601 25803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.390858 25803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.390885 25803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.391959 25803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.391993 25803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.392019 25803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:36:52.393417 25803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:33041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=36483
--tserver_master_addrs=127.10.109.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:36:52.393594 25803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.393757 25803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.398583 25811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.398653 25803 server_base.cc:1034] running on GCE node
W20260620 18:36:52.398602 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.398605 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.399098 25803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.399333 25803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.400490 25803 hybrid_clock.cc:648] HybridClock initialized: now 1781980612400485 us; error 36 us; skew 500 ppm
Jun 20 18:36:52 dist-test-slave-3jql krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980612, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:52.402978 25803 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:36:52.403386 25803 webserver.cc:460] Webserver started at http://127.10.109.67:36483/ using document root <none> and password file <none>
I20260620 18:36:52.403630 25803 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.403699 25803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.404978 25803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:52.405543 25818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.405716 25803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:36:52.405817 25803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "9221c3d4099d44f09bc3af270588a73a"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-3jql"
server_key: "f67c1d9779b6cdf81c55da0d673e293d"
server_key_iv: "37e28b49536d242218bdc8afb514e62a"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.406128 25803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.423631 25803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.425508 25803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.425655 25803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.425866 25803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.426149 25803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:52.426209 25803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.426251 25803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:52.426280 25803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.432322 25803 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33041
I20260620 18:36:52.432377 25931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33041 every 8 connection(s)
I20260620 18:36:52.432736 25803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:52.433727 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.433372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:40429)
0620 18:36:52.433514 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.433589 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 18:36:52.434003 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 25803
W20260620 18:36:52.434172 25932 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:35817 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:35817: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
I20260620 18:36:52.434494 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.434363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:33789)
0620 18:36:52.434434 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.434464 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:36:52.434907 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.434826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:34557)
0620 18:36:52.434857 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.434884 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 18:36:52.435053 25932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:36:53.247510 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.247279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:50989)
0620 18:36:53.247416 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.247451 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
I20260620 18:36:53.310909 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.310538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:37403)
0620 18:36:53.310731 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.310835 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260620 18:36:53.435564 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.435350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:34485)
0620 18:36:53.435459 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.435496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260620 18:36:53.985168 25939 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:53.985343 25939 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:53.985739 25939 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277), 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
Jun 20 18:36:53 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980612, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.67@KRBTEST.COM
Jun 20 18:36:53 dist-test-slave-3jql krb5kdc[24370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980612, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.66@KRBTEST.COM
I20260620 18:36:54.002743 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.986233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33041 (local address 127.10.109.65:60025)
0620 18:36:53.986350 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.986361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.986412 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.987844 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.987847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.988357 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.988634 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.988642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.990614 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.990615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.991498 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.991500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.992157 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.992158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.992261 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.996290 (+ 4029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:53.996294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.998423 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:54.000884 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:54.000892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:54.000895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:54.001190 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:54.001466 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:54.001467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:54.001467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:54.001622 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:54.002242 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:54.002250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:54.002399 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:36:54.002482 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:36:54.002611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":7,"thread_start_us":54,"threads_started":1}
I20260620 18:36:54.003062 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.986421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:60025 (local address 127.10.109.67:33041)
0620 18:36:53.986615 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.986626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.986660 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:36:53.987497 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.987501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.987576 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.987748 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.988834 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.990432 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.991605 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.992046 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.992185 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.998601 (+ 6416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.998605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.998606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:53.998619 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:54.000656 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:54.001283 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:54.001286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:54.001287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:54.001384 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:54.001734 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:54.001736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:54.001736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:54.002140 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:54.002255 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:54.002576 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:36:54.002737 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260620 18:36:54.003180 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.986345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:38651 (local address 127.10.109.66:34277)
0620 18:36:53.986606 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.986626 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.986660 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:36:53.987498 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.987502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.987575 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.987748 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.988838 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.990432 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.991626 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.992033 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.992142 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.998965 (+ 6823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.998969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.998969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:53.998978 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:54.001497 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:54.001869 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:54.001873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:54.001873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:54.002015 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:54.002280 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:54.002280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:54.002281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:54.002577 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:54.002678 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:54.002886 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:36:54.003055 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260620 18:36:54.003634 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.985982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:34277 (local address 127.10.109.65:38651)
0620 18:36:53.986100 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.986113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.986169 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.987844 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.987847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.988356 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.988634 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.988642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.990597 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.990598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.991499 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.991502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.992130 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.992131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.992262 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.996273 (+ 4011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:53.996282 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.998838 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:54.001590 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:54.001590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:54.001591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:54.001785 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:54.002113 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:54.002113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:54.002114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:54.002173 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:54.002663 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:54.002664 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:54.002702 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:36:54.002813 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:36:54.003525 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":78,"thread_start_us":84,"threads_started":1}
W20260620 18:36:54.004118 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
W20260620 18:36:54.004443 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:54.004492 25554 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0202ba269f4a8da1a1299436d7d87e; no voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f
I20260620 18:36:54.004637 25939 raft_consensus.cc:2726] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:36:54.248309 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.248171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:45121)
0620 18:36:54.248229 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.248269 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:36:54.311537 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.311304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:54677)
0620 18:36:54.311448 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.311491 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 18:36:54.436183 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.435988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:34595)
0620 18:36:54.436103 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.436143 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260620 18:36:55.248950 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.248754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:56609)
0620 18:36:55.248874 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:55.248906 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260620 18:36:55.312175 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.312000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:54295)
0620 18:36:55.312108 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:55.312140 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260620 18:36:55.436942 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.436712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:42371)
0620 18:36:55.436867 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:55.436903 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260620 18:36:56.196492 25949 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:56.196609 25949 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:56.196779 25949 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277), 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
W20260620 18:36:56.197253 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
W20260620 18:36:56.197321 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:56.197338 25554 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0202ba269f4a8da1a1299436d7d87e; no voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f
I20260620 18:36:56.197438 25949 raft_consensus.cc:2726] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:36:56.249748 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.249443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:59137)
0620 18:36:56.249587 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:56.249641 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260620 18:36:56.312767 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.312555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:42875)
0620 18:36:56.312679 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:56.312711 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260620 18:36:56.437683 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.437471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:58963)
0620 18:36:56.437599 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:56.437635 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260620 18:36:57.250392 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.250191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:44747)
0620 18:36:57.250312 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.250347 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260620 18:36:57.313387 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.313210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:60827)
0620 18:36:57.313317 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.313349 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260620 18:36:57.438469 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.438278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:35199)
0620 18:36:57.438399 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.438431 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260620 18:36:58.251091 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.250911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:55705)
0620 18:36:58.251014 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:58.251047 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260620 18:36:58.314018 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.313824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:54307)
0620 18:36:58.313950 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:58.313983 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260620 18:36:58.439080 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.438899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:44301)
0620 18:36:58.439011 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:58.439045 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260620 18:36:58.622632 25959 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:58.622772 25959 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:36:58.622946 25959 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277), 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
W20260620 18:36:58.623364 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
W20260620 18:36:58.623431 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:36:58.623466 25554 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0202ba269f4a8da1a1299436d7d87e; no voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f
I20260620 18:36:58.623548 25959 raft_consensus.cc:2726] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:36:59.251715 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.251505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:56297)
0620 18:36:59.251631 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.251667 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260620 18:36:59.314695 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.314510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:55517)
0620 18:36:59.314626 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.314659 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260620 18:36:59.439699 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.439514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:34655)
0620 18:36:59.439631 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.439665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 18:37:00.252373 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.252171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:34633)
0620 18:37:00.252286 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.252327 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260620 18:37:00.315397 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.315180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:43379)
0620 18:37:00.315322 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.315356 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260620 18:37:00.440321 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.440129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:49031)
0620 18:37:00.440244 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.440275 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 18:37:01.253137 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.252922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:51949)
0620 18:37:01.253051 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.253090 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260620 18:37:01.316125 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.315917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:52979)
0620 18:37:01.316045 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.316082 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260620 18:37:01.441051 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.440836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:50333)
0620 18:37:01.440975 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.441010 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260620 18:37:02.253890 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:02.253632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:47163)
0620 18:37:02.253797 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:02.253843 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260620 18:37:02.286445 25970 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:02.286608 25970 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:37:02.286847 25970 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277), 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
W20260620 18:37:02.287259 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
W20260620 18:37:02.287343 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:37:02.287374 25554 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0202ba269f4a8da1a1299436d7d87e; no voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f
I20260620 18:37:02.287447 25970 raft_consensus.cc:2726] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:02.316885 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:02.316648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:38009)
0620 18:37:02.316785 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:02.316833 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260620 18:37:02.441823 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:02.441579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:38649)
0620 18:37:02.441713 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:02.441762 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260620 18:37:03.254510 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.254320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:53303)
0620 18:37:03.254435 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.254466 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260620 18:37:03.317533 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.317326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:45487)
0620 18:37:03.317464 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.317497 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260620 18:37:03.442519 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.442323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:44495)
0620 18:37:03.442447 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.442480 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 18:37:04.255182 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.254982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:47671)
0620 18:37:04.255101 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.255137 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260620 18:37:04.318178 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.317976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:58617)
0620 18:37:04.318107 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.318141 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260620 18:37:04.443176 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.442965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:50341)
0620 18:37:04.443100 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.443133 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260620 18:37:05.255976 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.255674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:40665)
0620 18:37:05.255854 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.255889 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260620 18:37:05.318921 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.318645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:49055)
0620 18:37:05.318827 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.318878 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":146,"threads_started":1}
I20260620 18:37:05.443841 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.443638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:41747)
0620 18:37:05.443767 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.443800 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260620 18:37:06.256682 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.256487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:34945)
0620 18:37:06.256608 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:06.256637 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260620 18:37:06.319504 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.319304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:54843)
0620 18:37:06.319430 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:06.319461 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260620 18:37:06.444545 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.444331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:60993)
0620 18:37:06.444471 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:06.444503 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260620 18:37:06.927318 25985 raft_consensus.cc:482] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:06.927426 25985 raft_consensus.cc:504] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25bd70242c848b38498cccb2ddd050f" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 34277 } } peers { permanent_uuid: "9221c3d4099d44f09bc3af270588a73a" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33041 } } peers { permanent_uuid: "fb0202ba269f4a8da1a1299436d7d87e" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 33511 } }
I20260620 18:37:06.927568 25985 leader_election.cc:288] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277), 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041)
W20260620 18:37:06.928047 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9221c3d4099d44f09bc3af270588a73a (127.10.109.67:33041): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
W20260620 18:37:06.928166 25554 leader_election.cc:341] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f25bd70242c848b38498cccb2ddd050f (127.10.109.66:34277): Not found: Tablet not found: 79ea7212449c41e5a2b1dbeda8def2f9
I20260620 18:37:06.928205 25554 leader_election.cc:302] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0202ba269f4a8da1a1299436d7d87e; no voters: 9221c3d4099d44f09bc3af270588a73a, f25bd70242c848b38498cccb2ddd050f
I20260620 18:37:06.928282 25985 raft_consensus.cc:2726] T 79ea7212449c41e5a2b1dbeda8def2f9 P fb0202ba269f4a8da1a1299436d7d87e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:07.257438 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.257209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.65:39095)
0620 18:37:07.257349 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.257385 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260620 18:37:07.320197 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.320000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.66:56195)
0620 18:37:07.320117 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.320151 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 18:37:07.437733 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 25507
I20260620 18:37:07.442855 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 25670
I20260620 18:37:07.445353 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.445167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:35817 (local address 127.10.109.67:38317)
0620 18:37:07.445286 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.445318 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260620 18:37:07.447356 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 25803
2026-06-20T18:37:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18398 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:07 dist-test-slave-3jql krb5kdc[25991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:07 dist-test-slave-3jql krb5kdc[25991](info): set up 2 sockets
Jun 20 18:37:07 dist-test-slave-3jql krb5kdc[25991](info): commencing operation
krb5kdc: starting...
W20260620 18:37:08.491366 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:37:08 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:37:08.602504 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:08.680311 26004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:08.680495 26004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:08.680516 26004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:08.681737 26004 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:37:08.681773 26004 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:37:08.681787 26004 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:37:08.681798 26004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:08.681833 26004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:08.681861 26004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:08.681877 26004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:08.681902 26004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:08.683349 26004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:08.683588 26004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:08.683815 26004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:08.688482 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:08.688513 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:08.688513 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:08.688937 26004 server_base.cc:1034] running on GCE node
I20260620 18:37:08.689177 26004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:08.689448 26004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:08.690615 26004 hybrid_clock.cc:648] HybridClock initialized: now 1781980628690599 us; error 25 us; skew 500 ppm
Jun 20 18:37:08 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:08.693032 26004 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:37:08.693491 26004 webserver.cc:460] Webserver started at http://127.10.109.126:35989/ using document root <none> and password file <none>
I20260620 18:37:08.693713 26004 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:08.693779 26004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:08.693909 26004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:08.694985 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "2bdd574db784424d92f4796a2af2183b"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "06f5fac1990d0911dfecb724714d9252"
server_key_iv: "e4701b5eed05c4732b8db429e20795c6"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.695331 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "2bdd574db784424d92f4796a2af2183b"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "06f5fac1990d0911dfecb724714d9252"
server_key_iv: "e4701b5eed05c4732b8db429e20795c6"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.696574 26004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:08.697263 26019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:08.697468 26004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:37:08.697638 26004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "2bdd574db784424d92f4796a2af2183b"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "06f5fac1990d0911dfecb724714d9252"
server_key_iv: "e4701b5eed05c4732b8db429e20795c6"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.697723 26004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:08.715583 26004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:08.717386 26004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:08.717561 26004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:08.721899 26004 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:46501
I20260620 18:37:08.721927 26071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:46501 every 8 connection(s)
I20260620 18:37:08.722273 26004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:08.723068 26072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:08.725278 26072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap starting.
I20260620 18:37:08.725863 26072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:08.726168 26072 log.cc:824] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:08.726581 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26004
I20260620 18:37:08.726688 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:37:08.726927 10677 external_mini_cluster.cc:1428] Setting key 2cdfd0ebb327233bf5c69d0e5b67b878
I20260620 18:37:08.726935 26072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: No bootstrap required, opened a new log
I20260620 18:37:08.728647 26072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:08.728786 26072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:08.728809 26072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Initialized, Role: FOLLOWER
I20260620 18:37:08.728936 26072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:08.729034 26072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:08.729077 26072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:08.729147 26072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:08.729982 26072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:08.730103 26072 leader_election.cc:302] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bdd574db784424d92f4796a2af2183b; no voters:
I20260620 18:37:08.730289 26072 leader_election.cc:288] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:08.730343 26077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:08.730481 26077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 LEADER]: Becoming Leader. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Running, Role: LEADER
I20260620 18:37:08.730630 26072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:08.730615 26077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
W20260620 18:37:08.732746 26092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 18:37:08.731343 26078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:08.731365 26079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bdd574db784424d92f4796a2af2183b. Latest consensus state: current_term: 1 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:08.733373 26079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
W20260620 18:37:08.733316 26092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:37:08.733373 26078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:08.733672 26094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:08.734097 26094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:08.736349 26094 catalog_manager.cc:1347] Generated new cluster ID: 4eb4ec61df1946b38f45d4d1efe19050
I20260620 18:37:08.736444 26094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:37:08 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:08.741959 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.727410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.10.109.126:46501)
0620 18:37:08.727601 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:08.727611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:08.727646 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:37:08.728688 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:08.728695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:08.728747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:08.728915 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:08.729607 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.730836 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.733846 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.734503 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.734629 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.738009 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:08.738014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:08.738014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:08.738026 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:08.740230 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:08.740640 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:08.740643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:08.740644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:08.740765 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:08.741135 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:08.741136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:08.741137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:08.741405 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:08.741579 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:08.741656 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:37:08.741793 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 18:37:08.742055 26094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:08.742640 26094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:08.748270 26094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Generated new TSK 0
I20260620 18:37:08.748430 26094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:37:08.775276 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:08.856178 26098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:08.856402 26098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:08.856438 26098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:08.857547 26098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:08.857614 26098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:08.857666 26098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:37:08.859169 26098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:08.859418 26098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:08.859638 26098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:08.865064 26106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:08.865254 26098 server_base.cc:1034] running on GCE node
W20260620 18:37:08.865063 26104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:08.865106 26103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:08.865586 26098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:08.865833 26098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:08.867000 26098 hybrid_clock.cc:648] HybridClock initialized: now 1781980628866982 us; error 46 us; skew 500 ppm
Jun 20 18:37:08 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:08.869540 26098 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:37:08.869905 26098 webserver.cc:460] Webserver started at http://127.10.109.65:34071/ using document root <none> and password file <none>
I20260620 18:37:08.870132 26098 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:08.870180 26098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:08.870296 26098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:08.871352 26098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "51b0ba22bc4949f093afacba862a9105"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "3ce37b581fac504e7b965909f8e98a98"
server_key_iv: "fc916bf69f1edfc2196681a43bf7420b"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.871701 26098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "51b0ba22bc4949f093afacba862a9105"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "3ce37b581fac504e7b965909f8e98a98"
server_key_iv: "fc916bf69f1edfc2196681a43bf7420b"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.872973 26098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:37:08.873834 26113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:08.874001 26098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:37:08.874097 26098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "51b0ba22bc4949f093afacba862a9105"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "3ce37b581fac504e7b965909f8e98a98"
server_key_iv: "fc916bf69f1edfc2196681a43bf7420b"
server_key_version: "encryptionkey@0"
I20260620 18:37:08.874183 26098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:08.898600 26098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:08.900698 26098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:08.900853 26098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:08.901073 26098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:08.901387 26098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:08.901448 26098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:08.901491 26098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:08.901521 26098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:08.907131 26098 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:40315
I20260620 18:37:08.907231 26226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:40315 every 8 connection(s)
I20260620 18:37:08.907544 26098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:08.909862 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26098
I20260620 18:37:08.909950 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:08.910158 10677 external_mini_cluster.cc:1428] Setting key 16c9517235867a6451bc7323d2c3a0b2
Jun 20 18:37:08 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:08.917619 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.908648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:59153 (local address 127.10.109.126:46501)
0620 18:37:08.908744 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:08.908746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:08.908857 (+ 111us) server_negotiation.cc:407] Connection header received
0620 18:37:08.909309 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:08.909310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:08.909319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:08.909366 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:08.910414 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.911158 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.912246 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.912446 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.912503 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.914548 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:08.914549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:08.914550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:08.914560 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:08.916228 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:08.916622 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:08.916623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:08.916624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:08.916688 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:08.916924 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:08.916925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:08.916925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:08.917066 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:08.917146 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:08.917433 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:37:08.917520 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:37:08.917650 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.908577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:59153)
0620 18:37:08.908744 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.908776 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:08.909230 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:08.909421 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:08.909425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:08.910040 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:08.910287 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:08.910292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.911256 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.911257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:08.912154 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:08.912157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.912531 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.912532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:08.912680 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.913133 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:08.913161 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:08.914456 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:08.916326 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:08.916332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:08.916333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:08.916537 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:08.916762 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:08.916762 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:08.916763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:08.916856 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:08.917161 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:08.917167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:08.917277 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 18:37:08.917370 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:37:08.917481 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260620 18:37:08.918290 26227 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:08.918409 26227 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:08.918629 26227 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
I20260620 18:37:08.919209 26036 ts_manager.cc:194] Registered new tserver with Master: 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315)
I20260620 18:37:08.919844 26036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:59153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:37:08.937067 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:09.025697 26232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:09.025981 26232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:09.026010 26232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:09.027608 26232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:09.027663 26232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:09.027710 26232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:37:09.029805 26232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:09.030073 26232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:09.030326 26232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:09.037315 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:09.037262 26238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:09.037487 26232 server_base.cc:1034] running on GCE node
W20260620 18:37:09.037521 26237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:09.037758 26232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:09.038038 26232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:09.039208 26232 hybrid_clock.cc:648] HybridClock initialized: now 1781980629039177 us; error 45 us; skew 500 ppm
Jun 20 18:37:09 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:09.042502 26232 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:37:09.043020 26232 webserver.cc:460] Webserver started at http://127.10.109.66:39169/ using document root <none> and password file <none>
I20260620 18:37:09.043286 26232 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:09.043349 26232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:09.043484 26232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:09.044646 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "a267a3a414b64da393db4adc018f39d5"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "2c2c3fa08c633ceacb91942f7adad5ea"
server_key_iv: "8ac89c3ec8dabc632b71f0bd111a9925"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.045035 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "a267a3a414b64da393db4adc018f39d5"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "2c2c3fa08c633ceacb91942f7adad5ea"
server_key_iv: "8ac89c3ec8dabc632b71f0bd111a9925"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.047104 26232 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:37:09.050843 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.051422 26232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 18:37:09.051512 26232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "a267a3a414b64da393db4adc018f39d5"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "2c2c3fa08c633ceacb91942f7adad5ea"
server_key_iv: "8ac89c3ec8dabc632b71f0bd111a9925"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.051582 26232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:09.072327 26232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:09.074462 26232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:09.074565 26232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:09.074803 26232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:09.075065 26232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:09.075093 26232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.075114 26232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:09.075127 26232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.080823 26232 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37147
I20260620 18:37:09.080978 26360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37147 every 8 connection(s)
I20260620 18:37:09.081174 26232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:09.081619 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26232
I20260620 18:37:09.081743 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:09.081944 10677 external_mini_cluster.cc:1428] Setting key 0606158aa64916c0e1bbbe0550f0ffc0
Jun 20 18:37:09 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
I20260620 18:37:09.091477 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.081953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:32831 (local address 127.10.109.126:46501)
0620 18:37:09.081999 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.082001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.082305 (+ 304us) server_negotiation.cc:407] Connection header received
0620 18:37:09.082953 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.082954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.082967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.083052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:09.083894 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.084555 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.085659 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.085933 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.086007 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.088471 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:09.088472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:09.088472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:09.088480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:09.089950 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.090413 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.090414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.090415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.090479 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.090785 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.090786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.090787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.090910 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:09.090998 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.091284 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.091380 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:37:09.091679 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.081935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:32831)
0620 18:37:09.082174 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.082225 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.082851 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.083098 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.083101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.083620 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:09.083796 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.083801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.084667 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.084669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.085540 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.085543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.086035 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.086037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.086197 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.086657 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:09.086667 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:09.088343 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:09.090072 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:09.090079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:09.090081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:09.090314 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:09.090572 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:09.090573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:09.090574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:09.090688 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:09.090988 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:09.090995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:09.091144 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.091261 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.091422 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260620 18:37:09.092202 26361 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:09.092284 26361 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:09.092461 26361 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
I20260620 18:37:09.092849 26036 ts_manager.cc:194] Registered new tserver with Master: a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147)
I20260620 18:37:09.093269 26036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:32831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:37:09.109365 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:09.187776 26366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:09.187986 26366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:09.188022 26366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:09.189107 26366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:09.189172 26366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:09.189222 26366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:37:09.190598 26366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:09.190862 26366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:09.191077 26366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:09.196065 26371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:09.196075 26372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:09.196076 26374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:09.196360 26366 server_base.cc:1034] running on GCE node
I20260620 18:37:09.196566 26366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:09.196807 26366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:09.197973 26366 hybrid_clock.cc:648] HybridClock initialized: now 1781980629197972 us; error 41 us; skew 500 ppm
Jun 20 18:37:09 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:09.200521 26366 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:37:09.200980 26366 webserver.cc:460] Webserver started at http://127.10.109.67:45347/ using document root <none> and password file <none>
I20260620 18:37:09.201218 26366 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:09.201285 26366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:09.201418 26366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:09.202423 26366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "3709fb58fd604b2eaf7c8874db031a47"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "4d3fa3496d6e4f1a2f6c1b826bf6d36a"
server_key_iv: "e0832b1c5311563bd496a6c9c3ccb30b"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.202819 26366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "3709fb58fd604b2eaf7c8874db031a47"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "4d3fa3496d6e4f1a2f6c1b826bf6d36a"
server_key_iv: "e0832b1c5311563bd496a6c9c3ccb30b"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.204227 26366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:09.204957 26381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.205142 26366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:37:09.205240 26366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "3709fb58fd604b2eaf7c8874db031a47"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "4d3fa3496d6e4f1a2f6c1b826bf6d36a"
server_key_iv: "e0832b1c5311563bd496a6c9c3ccb30b"
server_key_version: "encryptionkey@0"
I20260620 18:37:09.205345 26366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:09.223234 26366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:09.225189 26366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:09.225369 26366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:09.225602 26366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:09.225919 26366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:09.225973 26366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.226008 26366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:09.226035 26366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:09.232707 26366 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33613
I20260620 18:37:09.232779 26494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33613 every 8 connection(s)
I20260620 18:37:09.233210 26366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:09.233662 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26366
I20260620 18:37:09.233767 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:09.233951 10677 external_mini_cluster.cc:1428] Setting key 6715896347446530054631a841dcf940
Jun 20 18:37:09 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:09.242534 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.234058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:45711 (local address 127.10.109.126:46501)
0620 18:37:09.234109 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.234113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.234346 (+ 233us) server_negotiation.cc:407] Connection header received
0620 18:37:09.234956 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.234957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.234966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.235008 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:09.235899 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.236388 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.237442 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.237639 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.237704 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.239850 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:09.239852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:09.239852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:09.239858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:09.241253 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.241678 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.241679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.241679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.241730 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.241946 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.241947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.241947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.242094 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:09.242165 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.242411 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.242471 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:37:09.242734 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.234065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:45711)
0620 18:37:09.234237 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.234277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.234863 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.235065 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.235068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.235615 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:09.235799 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.235804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.236535 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.236536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.237321 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.237323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.237722 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.237723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.237833 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.238274 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:09.238282 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:09.239758 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:09.241377 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:09.241390 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:09.241392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:09.241608 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:09.241791 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:09.241791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:09.241791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:09.241882 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:09.242163 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:09.242168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:09.242276 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.242362 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.242477 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260620 18:37:09.243311 26495 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:09.243403 26495 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:09.243605 26495 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
I20260620 18:37:09.244012 26036 ts_manager.cc:194] Registered new tserver with Master: 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
I20260620 18:37:09.244403 26036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:45711
I20260620 18:37:09.245774 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:09.253595 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.247173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.10.109.126:46501)
0620 18:37:09.247244 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.247246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.247361 (+ 115us) server_negotiation.cc:407] Connection header received
0620 18:37:09.247438 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.247438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.247448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.247522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:09.248081 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.248658 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.249455 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.249654 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.249822 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.251092 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:09.251093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:09.251094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:09.251100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:09.252441 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.252784 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.252785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.252786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.252856 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:09.253056 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:09.253057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:09.253058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:09.253207 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:09.253307 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.253418 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.253499 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:37:09.255244 26036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:09.256951 26036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:09.263329 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.0.0.1:49806)
0620 18:37:09.259543 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.259584 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.259646 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.260136 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.260137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.260139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:09.260168 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.260169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.261062 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.261063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.262071 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.262073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.263014 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.263107 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.263121 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.263202 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.263251 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"spinlock_wait_cycles":384,"thread_start_us":86,"threads_started":1}
I20260620 18:37:09.263350 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.10.109.65:40315)
0620 18:37:09.259834 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.259840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.259850 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:37:09.259889 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.259891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.259906 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.260235 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:09.260291 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.260962 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.262158 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.262930 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263006 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.263070 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.263224 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.263266 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":70,"threads_started":1}
I20260620 18:37:09.263844 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37147 (local address 127.0.0.1:44114)
0620 18:37:09.259668 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.259680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.259734 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.259964 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.259967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.259970 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:09.260037 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.260039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.261619 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.261620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.262525 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.262527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.263678 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.263734 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.263737 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.263772 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.263801 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260620 18:37:09.264027 26161 tablet_service.cc:1467] Processing CreateTablet for tablet dc68a4ab76874c038938920a78ea4cd3 (DEFAULT_TABLE table=test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:09.264225 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.10.109.67:33613)
0620 18:37:09.259522 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.259527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.259867 (+ 340us) server_negotiation.cc:407] Connection header received
0620 18:37:09.260171 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.260174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.260189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.260284 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:09.260423 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.261509 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263069 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.263782 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263828 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.263875 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.264095 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.264139 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260620 18:37:09.264237 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.10.109.66:37147)
0620 18:37:09.259805 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.259809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.259820 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:37:09.259858 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.259860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.259877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.259954 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:09.260367 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.261507 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.262630 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.263307 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263371 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.264066 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.264092 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.264135 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260620 18:37:09.264433 26161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc68a4ab76874c038938920a78ea4cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:09.264446 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.259276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33613 (local address 127.0.0.1:55088)
0620 18:37:09.259777 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.259786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.259991 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.260281 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.260282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.260284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:09.260334 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.260336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.261724 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.261725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.262978 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.262981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.263935 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.263936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.263996 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.263999 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.264327 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.264375 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":111,"threads_started":1}
I20260620 18:37:09.264902 26295 tablet_service.cc:1467] Processing CreateTablet for tablet dc68a4ab76874c038938920a78ea4cd3 (DEFAULT_TABLE table=test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:09.265027 26429 tablet_service.cc:1467] Processing CreateTablet for tablet dc68a4ab76874c038938920a78ea4cd3 (DEFAULT_TABLE table=test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:09.265211 26295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc68a4ab76874c038938920a78ea4cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:09.265348 26429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc68a4ab76874c038938920a78ea4cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:09.267786 26512 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Bootstrap starting.
I20260620 18:37:09.268498 26514 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Bootstrap starting.
I20260620 18:37:09.268632 26512 tablet_bootstrap.cc:654] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:09.268746 26513 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Bootstrap starting.
I20260620 18:37:09.269158 26514 tablet_bootstrap.cc:654] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:09.269169 26512 log.cc:824] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:09.269380 26513 tablet_bootstrap.cc:654] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:09.269775 26514 log.cc:824] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:09.269889 26512 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: No bootstrap required, opened a new log
I20260620 18:37:09.269953 26512 ts_tablet_manager.cc:1397] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:37:09.269984 26513 log.cc:824] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:09.270656 26514 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: No bootstrap required, opened a new log
I20260620 18:37:09.270738 26514 ts_tablet_manager.cc:1397] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:37:09.270889 26513 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: No bootstrap required, opened a new log
I20260620 18:37:09.270991 26513 ts_tablet_manager.cc:1397] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:37:09.272115 26514 raft_consensus.cc:348] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.272109 26512 raft_consensus.cc:348] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.272256 26514 raft_consensus.cc:374] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:09.272264 26512 raft_consensus.cc:374] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:09.272280 26514 raft_consensus.cc:729] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3709fb58fd604b2eaf7c8874db031a47, State: Initialized, Role: FOLLOWER
I20260620 18:37:09.272315 26512 raft_consensus.cc:729] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a267a3a414b64da393db4adc018f39d5, State: Initialized, Role: FOLLOWER
I20260620 18:37:09.272414 26514 consensus_queue.cc:260] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.272449 26512 consensus_queue.cc:260] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.272691 26514 ts_tablet_manager.cc:1428] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:37:09.272698 26512 ts_tablet_manager.cc:1428] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:09.272819 26495 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
I20260620 18:37:09.272843 26361 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
I20260620 18:37:09.273204 26513 raft_consensus.cc:348] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.273340 26513 raft_consensus.cc:374] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:09.273399 26513 raft_consensus.cc:729] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b0ba22bc4949f093afacba862a9105, State: Initialized, Role: FOLLOWER
I20260620 18:37:09.273504 26513 consensus_queue.cc:260] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.273798 26513 ts_tablet_manager.cc:1428] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:09.273825 26227 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
W20260620 18:37:09.331768 26362 tablet.cc:2369] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:09.380939 26519 raft_consensus.cc:482] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:09.381101 26519 raft_consensus.cc:504] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.381469 26519 leader_election.cc:288] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
I20260620 18:37:09.384454 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.381719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:45135 (local address 127.10.109.67:33613)
0620 18:37:09.381766 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.381769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.381778 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:37:09.381810 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.381813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.381824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.381897 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:09.382078 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.382633 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.383679 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.384077 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.384141 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.384148 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.384353 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.384386 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:37:09.384465 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.381726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:40259 (local address 127.10.109.65:40315)
0620 18:37:09.381766 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.381769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.381961 (+ 192us) server_negotiation.cc:407] Connection header received
0620 18:37:09.381994 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.381995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.382005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.382060 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:09.382215 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.382658 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.383758 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.384137 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.384209 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.384218 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.384363 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.384399 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:37:09.384467 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.381642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33613 (local address 127.10.109.66:45135)
0620 18:37:09.381699 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.381711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.381754 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.381906 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.381907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.381908 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:09.381998 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.381999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.382713 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.382715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.383594 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.383597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.384162 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.384164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.384261 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.384265 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.384341 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.384386 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260620 18:37:09.384470 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.381643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.10.109.66:40259)
0620 18:37:09.381879 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.381889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:09.381947 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:09.382076 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:09.382076 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:09.382078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:09.382149 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.382150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.382794 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.382795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.383662 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:09.383665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.384220 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.384221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:09.384271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.384274 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:37:09.384339 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:37:09.384377 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":13,"thread_start_us":153,"threads_started":1}
I20260620 18:37:09.384852 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc68a4ab76874c038938920a78ea4cd3" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3709fb58fd604b2eaf7c8874db031a47" is_pre_election: true
I20260620 18:37:09.385007 26449 raft_consensus.cc:2454] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 0.
I20260620 18:37:09.385027 26181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc68a4ab76874c038938920a78ea4cd3" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b0ba22bc4949f093afacba862a9105" is_pre_election: true
I20260620 18:37:09.385171 26181 raft_consensus.cc:2454] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 0.
I20260620 18:37:09.385237 26249 leader_election.cc:302] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5; no voters:
I20260620 18:37:09.385453 26519 raft_consensus.cc:2781] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:09.385506 26519 raft_consensus.cc:482] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:09.385522 26519 raft_consensus.cc:3037] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:09.386240 26519 raft_consensus.cc:504] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.386385 26519 leader_election.cc:288] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 election: Requested vote from peers 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
I20260620 18:37:09.386508 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc68a4ab76874c038938920a78ea4cd3" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3709fb58fd604b2eaf7c8874db031a47"
I20260620 18:37:09.386520 26181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc68a4ab76874c038938920a78ea4cd3" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b0ba22bc4949f093afacba862a9105"
I20260620 18:37:09.386612 26181 raft_consensus.cc:3037] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:09.386587 26449 raft_consensus.cc:3037] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:09.387326 26181 raft_consensus.cc:2454] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 1.
I20260620 18:37:09.387555 26248 leader_election.cc:302] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105, a267a3a414b64da393db4adc018f39d5; no voters:
I20260620 18:37:09.387603 26449 raft_consensus.cc:2454] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 1.
I20260620 18:37:09.387676 26519 raft_consensus.cc:2781] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:09.387864 26519 raft_consensus.cc:686] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 1 LEADER]: Becoming Leader. State: Replica: a267a3a414b64da393db4adc018f39d5, State: Running, Role: LEADER
I20260620 18:37:09.387984 26519 consensus_queue.cc:237] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:09.388855 26036 catalog_manager.cc:5495] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a267a3a414b64da393db4adc018f39d5 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "a267a3a414b64da393db4adc018f39d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } health_report { overall_health: UNKNOWN } } }
W20260620 18:37:09.408420 26228 tablet.cc:2369] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:09.411379 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.409009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.10.109.66:37147)
0620 18:37:09.409074 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:09.409076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:09.409090 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:37:09.409187 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:09.409187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:09.409217 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:09.409278 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:09.409409 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.409883 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.410594 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:09.410794 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:09.410845 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:09.411007 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:09.411104 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:09.411182 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:09.411253 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:37:09.411271 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:37:09.413625 26181 raft_consensus.cc:1261] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Refusing update from remote peer a267a3a414b64da393db4adc018f39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:09.413645 26449 raft_consensus.cc:1261] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Refusing update from remote peer a267a3a414b64da393db4adc018f39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:09.413983 26522 consensus_queue.cc:1035] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:09.414125 26519 consensus_queue.cc:1035] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:09.416424 26525 mvcc.cc:205] Tried to move back new op lower bound from 7298992658074533888 to 7298992657973895168. Current Snapshot: MvccSnapshot[applied={T|T < 7298992658074533888}]
I20260620 18:37:09.416541 26526 mvcc.cc:205] Tried to move back new op lower bound from 7298992658074533888 to 7298992657973895168. Current Snapshot: MvccSnapshot[applied={T|T < 7298992658074533888}]
I20260620 18:37:09.416879 26527 mvcc.cc:205] Tried to move back new op lower bound from 7298992658074533888 to 7298992657973895168. Current Snapshot: MvccSnapshot[applied={T|T < 7298992658074533888}]
I20260620 18:37:09.418913 26036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:09.419025 26036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:09.419790 26036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Sending DeleteTablet for 3 replicas of tablet dc68a4ab76874c038938920a78ea4cd3
I20260620 18:37:09.420123 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26098
I20260620 18:37:09.420111 26295 tablet_service.cc:1514] Processing DeleteTablet for tablet dc68a4ab76874c038938920a78ea4cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:09 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:44114
I20260620 18:37:09.420157 26161 tablet_service.cc:1514] Processing DeleteTablet for tablet dc68a4ab76874c038938920a78ea4cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:09 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:49806
I20260620 18:37:09.420878 26429 tablet_service.cc:1514] Processing DeleteTablet for tablet dc68a4ab76874c038938920a78ea4cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:09 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55088
I20260620 18:37:09.421087 26535 tablet_replica.cc:330] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: stopping tablet replica
I20260620 18:37:09.421178 26535 raft_consensus.cc:2229] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:09.421306 26535 raft_consensus.cc:2258] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:09.421445 26536 tablet_replica.cc:330] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: stopping tablet replica
I20260620 18:37:09.421535 26536 raft_consensus.cc:2229] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:09.421633 26536 raft_consensus.cc:2258] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:09.422142 26536 ts_tablet_manager.cc:1905] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:09.422149 26535 ts_tablet_manager.cc:1905] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:09.424008 26536 ts_tablet_manager.cc:1918] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:09.424142 26536 log.cc:1196] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/dc68a4ab76874c038938920a78ea4cd3
I20260620 18:37:09.424178 26535 ts_tablet_manager.cc:1918] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:09.424263 26535 log.cc:1196] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/dc68a4ab76874c038938920a78ea4cd3
I20260620 18:37:09.424525 26535 ts_tablet_manager.cc:1939] T dc68a4ab76874c038938920a78ea4cd3 P a267a3a414b64da393db4adc018f39d5: Deleting consensus metadata
I20260620 18:37:09.424824 26536 ts_tablet_manager.cc:1939] T dc68a4ab76874c038938920a78ea4cd3 P 3709fb58fd604b2eaf7c8874db031a47: Deleting consensus metadata
I20260620 18:37:09.424888 26023 catalog_manager.cc:4841] TS a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): tablet dc68a4ab76874c038938920a78ea4cd3 (table test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]) successfully deleted
I20260620 18:37:09.425338 26021 catalog_manager.cc:4841] TS 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): tablet dc68a4ab76874c038938920a78ea4cd3 (table test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]) successfully deleted
W20260620 18:37:09.426623 26020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:40315 (error 108)
I20260620 18:37:09.427152 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:09.426999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.0.0.1:49808)
0620 18:37:09.427065 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:09.427109 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:37:09.427230 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26232
W20260620 18:37:09.427269 26020 catalog_manager.cc:4584] TS 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc68a4ab76874c038938920a78ea4cd3: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40315: connect: Connection refused (error 111)
I20260620 18:37:09.431572 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26366
I20260620 18:37:09.435912 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26004
I20260620 18:37:29.441110 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--webserver_interface=127.10.109.126
--webserver_port=35989
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:29.521266 26538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:29.521452 26538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:29.521473 26538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:29.522516 26538 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:37:29.522547 26538 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:37:29.522560 26538 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:37:29.522572 26538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:29.522584 26538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:29.522598 26538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:29.522609 26538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:29.522620 26538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:29.523988 26538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:29.524168 26538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:29.524324 26538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:29.532697 26547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.532703 26545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.532788 26544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.532922 26538 server_base.cc:1034] running on GCE node
I20260620 18:37:29.533180 26538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:29.533423 26538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:29.534560 26538 hybrid_clock.cc:648] HybridClock initialized: now 1781980649534550 us; error 33 us; skew 500 ppm
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:29.537119 26538 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:37:29.537570 26538 webserver.cc:460] Webserver started at http://127.10.109.126:35989/ using document root <none> and password file <none>
I20260620 18:37:29.537811 26538 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:29.537868 26538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:29.539412 26538 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:29.540156 26554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.540335 26538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:37:29.540444 26538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "2bdd574db784424d92f4796a2af2183b"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "06f5fac1990d0911dfecb724714d9252"
server_key_iv: "e4701b5eed05c4732b8db429e20795c6"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.540830 26538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:29.557823 26538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:29.559736 26538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:29.559890 26538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:29.563990 26538 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:46501
I20260620 18:37:29.564023 26606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:46501 every 8 connection(s)
I20260620 18:37:29.564445 26538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:29.565949 26607 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:37:29.566105 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26538
I20260620 18:37:29.566390 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:40315
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=34071
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:29.566852 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap starting.
I20260620 18:37:29.576562 26607 log.cc:824] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:29.578612 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:37:29.578992 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap complete.
I20260620 18:37:29.581377 26607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:29.581630 26607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Initialized, Role: FOLLOWER
I20260620 18:37:29.581835 26607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:29.581946 26607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:29.582010 26607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:29.582062 26607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:37:29.583221 26607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:29.583346 26607 leader_election.cc:302] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bdd574db784424d92f4796a2af2183b; no voters:
I20260620 18:37:29.583530 26607 leader_election.cc:288] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:37:29.583595 26611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:37:29.583738 26611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 LEADER]: Becoming Leader. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Running, Role: LEADER
I20260620 18:37:29.583808 26607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:29.583829 26611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:29.584080 26611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bdd574db784424d92f4796a2af2183b. Latest consensus state: current_term: 2 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:29.584148 26611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:29.584082 26612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:29.584321 26615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:29.584396 26612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:29.585042 26615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:29.585161 26615 catalog_manager.cc:1259] Loaded cluster ID: 4eb4ec61df1946b38f45d4d1efe19050
I20260620 18:37:29.585210 26615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:29.586601 26615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:29.586820 26615 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Loaded TSK: 0
I20260620 18:37:29.597787 26615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Generated new TSK 1
I20260620 18:37:29.597869 26615 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:37:29.651877 26609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:29.652071 26609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:29.652094 26609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:29.653246 26609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:29.653286 26609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:29.653312 26609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:37:29.654897 26609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:40315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=34071
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:29.655120 26609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:29.655352 26609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:29.660475 26635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.660439 26634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.660466 26637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.660701 26609 server_base.cc:1034] running on GCE node
I20260620 18:37:29.661015 26609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:29.661307 26609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:29.662513 26609 hybrid_clock.cc:648] HybridClock initialized: now 1781980649662497 us; error 28 us; skew 500 ppm
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:29.665500 26609 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:37:29.666002 26609 webserver.cc:460] Webserver started at http://127.10.109.65:34071/ using document root <none> and password file <none>
I20260620 18:37:29.666246 26609 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:29.666302 26609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:29.667722 26609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:29.668483 26644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.668679 26609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:29.668776 26609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "51b0ba22bc4949f093afacba862a9105"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "3ce37b581fac504e7b965909f8e98a98"
server_key_iv: "fc916bf69f1edfc2196681a43bf7420b"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.669118 26609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:29.679773 26609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:29.682065 26609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:29.682246 26609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:29.682502 26609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:29.683018 26651 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:37:29.685314 26609 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:37:29.685376 26609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 18:37:29.685424 26609 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:37:29.686049 26609 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:37:29.686106 26609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:37:29.686179 26651 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Bootstrap starting.
I20260620 18:37:29.693480 26609 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:40315
I20260620 18:37:29.693611 26758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:40315 every 8 connection(s)
I20260620 18:37:29.693886 26609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:29.698238 26651 log.cc:824] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:29.701202 26651 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:37:29.701542 26651 tablet_bootstrap.cc:492] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Bootstrap complete.
I20260620 18:37:29.702472 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26609
I20260620 18:37:29.702785 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:37147
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=39169
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:29.703302 26651 ts_tablet_manager.cc:1397] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:29.705194 26651 raft_consensus.cc:348] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:29.705854 26651 raft_consensus.cc:729] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b0ba22bc4949f093afacba862a9105, State: Initialized, Role: FOLLOWER
I20260620 18:37:29.706046 26651 consensus_queue.cc:260] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } } peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } }
I20260620 18:37:29.706707 26651 ts_tablet_manager.cc:1428] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:29.710292 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.694662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:55677)
0620 18:37:29.694940 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:29.694985 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:29.695404 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:29.699034 (+ 3630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:29.699038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:29.699324 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:29.699498 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.699503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.700651 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.700653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.701542 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.701546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.702154 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.702155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.702299 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.703135 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:29.703144 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:29.705658 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:29.708252 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.708259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.708262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.708584 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.708904 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.708905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.708906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.709094 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.709581 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:29.709588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:29.709749 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:37:29.709841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:37:29.710024 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260620 18:37:29.710966 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.694772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:55677 (local address 127.10.109.126:46501)
0620 18:37:29.698506 (+ 3734us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:29.698515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:29.698601 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:37:29.698733 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:29.698786 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:29.698851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:29.698998 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:29.699621 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.700507 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.701678 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.702060 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.702179 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.705792 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:29.705797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:29.705797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:29.705808 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:29.708145 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.708690 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.708694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.708694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.708817 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.709181 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.709182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.709182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.709488 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:29.710456 (+ 968us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:29.710537 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:37:29.710688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3678,"thread_start_us":67,"threads_started":1}
I20260620 18:37:29.711690 26759 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:29.711791 26759 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:29.711974 26759 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
I20260620 18:37:29.712762 26571 ts_manager.cc:194] Registered new tserver with Master: 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315)
I20260620 18:37:29.713928 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:55677
I20260620 18:37:29.715008 26759 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.65@KRBTEST.COM
I20260620 18:37:29.723414 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.713424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.0.0.1:36882)
0620 18:37:29.714222 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:29.714250 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:29.714324 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:29.714575 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:29.714579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:29.714925 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:29.714989 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.714991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.716182 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.716183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.717085 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.717087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.717808 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.717810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.717902 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.718414 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:29.718419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:29.720032 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:29.722158 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.722165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.722167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.722412 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.722653 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.722653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.722654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.722715 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.723083 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:29.723089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:29.723141 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:37:29.723215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:37:29.723317 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":89,"threads_started":1}
I20260620 18:37:29.723493 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.713672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.10.109.65:40315)
0620 18:37:29.713819 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:29.713824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:29.714398 (+ 574us) server_negotiation.cc:407] Connection header received
0620 18:37:29.714435 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:29.714438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:29.714452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:29.714505 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:29.715089 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.716093 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.717173 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.717696 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.717789 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.720141 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:29.720148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:29.720149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:29.720156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:29.722063 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.722507 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.722510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.722511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.722577 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.722825 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.722826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.722827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.723007 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:29.723096 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:29.723289 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:37:29.723390 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260620 18:37:29.724165 26681 tablet_service.cc:1514] Processing DeleteTablet for tablet dc68a4ab76874c038938920a78ea4cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:09 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36882
I20260620 18:37:29.724786 26768 tablet_replica.cc:330] stopping tablet replica
I20260620 18:37:29.724918 26768 raft_consensus.cc:2229] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:29.724994 26768 raft_consensus.cc:2258] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:29.725383 26768 ts_tablet_manager.cc:1905] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:29.727890 26768 ts_tablet_manager.cc:1918] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:29.727962 26768 log.cc:1196] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal/wals/dc68a4ab76874c038938920a78ea4cd3
I20260620 18:37:29.728190 26768 ts_tablet_manager.cc:1939] T dc68a4ab76874c038938920a78ea4cd3 P 51b0ba22bc4949f093afacba862a9105: Deleting consensus metadata
I20260620 18:37:29.728705 26555 catalog_manager.cc:4841] TS 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315): tablet dc68a4ab76874c038938920a78ea4cd3 (table test-table [id=5fe6c0b2a99443f092d8989a4b2a8aab]) successfully deleted
W20260620 18:37:29.791301 26764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:29.791548 26764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:29.791587 26764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:29.792755 26764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:29.792820 26764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:29.792866 26764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:37:29.794333 26764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=39169
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:29.794602 26764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:29.794883 26764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:29.799954 26776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.799957 26774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.800170 26764 server_base.cc:1034] running on GCE node
W20260620 18:37:29.800304 26773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.800523 26764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:29.800791 26764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:29.801952 26764 hybrid_clock.cc:648] HybridClock initialized: now 1781980649801941 us; error 28 us; skew 500 ppm
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:29.804502 26764 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:37:29.804937 26764 webserver.cc:460] Webserver started at http://127.10.109.66:39169/ using document root <none> and password file <none>
I20260620 18:37:29.805130 26764 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:29.805181 26764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:29.806452 26764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:29.807054 26783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.807210 26764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:37:29.807296 26764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "a267a3a414b64da393db4adc018f39d5"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "2c2c3fa08c633ceacb91942f7adad5ea"
server_key_iv: "8ac89c3ec8dabc632b71f0bd111a9925"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.807615 26764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:29.844957 26764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:29.847342 26764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:29.847491 26764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:29.847690 26764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:29.848011 26764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:29.848045 26764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.848093 26764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:29.848119 26764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.853932 26764 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37147
I20260620 18:37:29.854010 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37147 every 8 connection(s)
I20260620 18:37:29.854308 26764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:29.859984 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26764
I20260620 18:37:29.860270 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:33613
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=45347
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:29.864356 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.855072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:50533 (local address 127.10.109.126:46501)
0620 18:37:29.855125 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:29.855127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:29.855346 (+ 219us) server_negotiation.cc:407] Connection header received
0620 18:37:29.855807 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:29.855808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:29.855821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:29.855890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:29.856905 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.857408 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.858409 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.858624 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.858690 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.861133 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:29.861135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:29.861135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:29.861142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:29.862718 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.863267 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.863269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.863269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.863325 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:29.863623 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:29.863624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:29.863624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:29.863733 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:29.863788 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:29.864199 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 18:37:29.864289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 18:37:29.864585 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.855082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:50533)
0620 18:37:29.855223 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:29.855257 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:29.855726 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:29.855922 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:29.855926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:29.856488 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:29.856784 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.856795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.857510 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.857511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.858316 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:29.858318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.858720 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.858721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:29.858878 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.859303 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:29.859311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:29.861025 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:29.862856 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.862862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.862864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.863176 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.863402 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:29.863403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:29.863404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:29.863552 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:29.863804 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:29.863810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:29.863979 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 18:37:29.864108 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:37:29.864290 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260620 18:37:29.865156 26897 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:29.865617 26897 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:29.865832 26897 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
I20260620 18:37:29.866319 26571 ts_manager.cc:194] Registered new tserver with Master: a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147)
I20260620 18:37:29.866727 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:50533
W20260620 18:37:29.944387 26900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:29.944831 26900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:29.944873 26900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:29.946867 26900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:29.946940 26900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:29.946990 26900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:37:29.948865 26900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:33613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=45347
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:29.949155 26900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:29.949419 26900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:29.954384 26905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.954411 26906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.954384 26908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.954857 26900 server_base.cc:1034] running on GCE node
I20260620 18:37:29.955114 26900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:29.955354 26900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:29.956522 26900 hybrid_clock.cc:648] HybridClock initialized: now 1781980649956523 us; error 43 us; skew 500 ppm
Jun 20 18:37:29 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:29.958992 26900 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:37:29.959347 26900 webserver.cc:460] Webserver started at http://127.10.109.67:45347/ using document root <none> and password file <none>
I20260620 18:37:29.959595 26900 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:29.959651 26900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:29.961009 26900 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:29.961673 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.961855 26900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:37:29.961943 26900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "3709fb58fd604b2eaf7c8874db031a47"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "4d3fa3496d6e4f1a2f6c1b826bf6d36a"
server_key_iv: "e0832b1c5311563bd496a6c9c3ccb30b"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.962304 26900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:29.993937 26900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:29.996366 26900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:29.996524 26900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:29.996769 26900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:29.997082 26900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:29.997138 26900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.997179 26900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:29.997207 26900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:30.004910 26900 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33613
I20260620 18:37:30.004956 27028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33613 every 8 connection(s)
I20260620 18:37:30.005569 26900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:30.006562 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 26900
Jun 20 18:37:30 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:37:30.015033 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.007011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.10.109.126:46501)
0620 18:37:30.007124 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.007126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.007234 (+ 108us) server_negotiation.cc:407] Connection header received
0620 18:37:30.007281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.007282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.007294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.007369 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:30.007973 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.008739 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.010068 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.010364 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.010448 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.012083 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:30.012085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:30.012086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:30.012092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:30.013796 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.014152 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.014153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.014153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.014215 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.014424 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.014425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.014425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.014543 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:30.014738 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.014796 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.014920 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260620 18:37:30.017221 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.006436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:52425)
0620 18:37:30.006603 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.006679 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.007398 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.007604 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:30.007613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:30.008288 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:30.008556 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.008562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.009826 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.009827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.011011 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.011014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.011395 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.011396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.011517 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.012066 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:30.012076 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:30.013817 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:30.015586 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:30.015592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:30.015594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:30.015892 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:30.016115 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:30.016115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:30.016116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:30.016211 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:30.016708 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:30.016713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:30.016829 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 18:37:30.016925 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:37:30.017041 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260620 18:37:30.017354 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.006474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:52425 (local address 127.10.109.126:46501)
0620 18:37:30.006796 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.006799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.006814 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:37:30.007473 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.007474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.007482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.007515 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:30.008738 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.009713 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.011099 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.011299 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.011388 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.013928 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:30.013929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:30.013930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:30.013935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:30.015529 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.015970 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.015971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.015972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.016031 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.016383 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.016385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.016386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.016591 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:30.016696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.017016 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.017243 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260620 18:37:30.017729 27029 heartbeater.cc:347] Connected to a master server at 127.10.109.126:46501
I20260620 18:37:30.017828 27029 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:30.017999 27029 heartbeater.cc:511] Master 127.10.109.126:46501 requested a full tablet report, sending...
I20260620 18:37:30.018327 26571 ts_manager.cc:194] Registered new tserver with Master: 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
I20260620 18:37:30.018734 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:52425
I20260620 18:37:30.026710 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:30.029681 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.027129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.10.109.126:46501)
0620 18:37:30.027182 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.027185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.027296 (+ 111us) server_negotiation.cc:407] Connection header received
0620 18:37:30.027383 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.027384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.027424 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.027509 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:30.027614 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.028063 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.028818 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.029025 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.029101 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.029277 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:30.029327 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:37:30.029618 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55266: token expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 18:37:30.029771 26762 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55266: token expired
W20260620 18:37:30.030416 26500 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.109.126:46501)
I20260620 18:37:30.039616 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.030930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.10.109.126:46501)
0620 18:37:30.030970 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.030972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.031040 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:37:30.031130 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.031131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.031139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.031234 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:30.032070 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.032910 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.034041 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.034383 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.034468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.035872 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:30.035874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:30.035875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:30.035881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:30.038251 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.038643 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.038644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.038645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.038723 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:30.039124 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:30.039125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:30.039125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:30.039243 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:30.039322 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.039460 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.039517 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:37:30.040934 26571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:30.042618 26571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:30.044791 26681 tablet_service.cc:1467] Processing CreateTablet for tablet e9a616d6838847f4a70428c340023ce0 (DEFAULT_TABLE table=test-table [id=6f9f6fc039fa44149f3271e78827a09c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:30.044997 26681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a616d6838847f4a70428c340023ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:30.046540 26651 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Bootstrap starting.
I20260620 18:37:30.046900 26651 tablet_bootstrap.cc:654] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:30.047525 26651 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: No bootstrap required, opened a new log
I20260620 18:37:30.047575 26651 ts_tablet_manager.cc:1397] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 18:37:30.047904 26651 raft_consensus.cc:348] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.047981 26651 raft_consensus.cc:374] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:30.048015 26651 raft_consensus.cc:729] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b0ba22bc4949f093afacba862a9105, State: Initialized, Role: FOLLOWER
I20260620 18:37:30.048064 26651 consensus_queue.cc:260] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.048182 26651 ts_tablet_manager.cc:1428] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 18:37:30.049405 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.044522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37147 (local address 127.0.0.1:52976)
0620 18:37:30.044564 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.044575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.044610 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.045086 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:30.045087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:30.045089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:30.045144 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.045145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.046320 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.046321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.047533 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.047536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.049183 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.049184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.049262 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.049282 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:37:30.049331 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:37:30.049361 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 18:37:30.049471 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.044732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.10.109.66:37147)
0620 18:37:30.044897 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.044901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.044911 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:37:30.044970 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.044972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.044988 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.045074 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:30.045213 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.046220 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.048348 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.049117 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.049169 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.049241 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.049356 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.049392 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260620 18:37:30.050246 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.044606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33613 (local address 127.0.0.1:58384)
0620 18:37:30.044700 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.044709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.044743 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.046912 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:30.046912 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:30.046914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:30.046956 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.046958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.048125 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.048126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.049134 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.049136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.050015 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.050016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.050082 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.050085 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:37:30.050141 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:37:30.050179 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260620 18:37:30.050169 26831 tablet_service.cc:1467] Processing CreateTablet for tablet e9a616d6838847f4a70428c340023ce0 (DEFAULT_TABLE table=test-table [id=6f9f6fc039fa44149f3271e78827a09c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:30.050262 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.044909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.10.109.67:33613)
0620 18:37:30.046623 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.046627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.046637 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:37:30.046678 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.046682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.046700 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.046810 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:30.047042 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.048056 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.049232 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.049944 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.050015 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.050065 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.050154 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.050189 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1680,"thread_start_us":76,"threads_started":1}
I20260620 18:37:30.050522 26831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a616d6838847f4a70428c340023ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:30.050922 26963 tablet_service.cc:1467] Processing CreateTablet for tablet e9a616d6838847f4a70428c340023ce0 (DEFAULT_TABLE table=test-table [id=6f9f6fc039fa44149f3271e78827a09c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:30.051232 26963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a616d6838847f4a70428c340023ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:30.054157 27039 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Bootstrap starting.
I20260620 18:37:30.054615 27040 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Bootstrap starting.
I20260620 18:37:30.054883 27039 tablet_bootstrap.cc:654] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:30.055220 27040 tablet_bootstrap.cc:654] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:30.055431 27039 log.cc:824] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:30.055787 27040 log.cc:824] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:30.056257 27039 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: No bootstrap required, opened a new log
I20260620 18:37:30.056422 27039 ts_tablet_manager.cc:1397] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:37:30.056689 27040 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: No bootstrap required, opened a new log
I20260620 18:37:30.056767 27040 ts_tablet_manager.cc:1397] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:37:30.058074 27039 raft_consensus.cc:348] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.058199 27039 raft_consensus.cc:374] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:30.058229 27039 raft_consensus.cc:729] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a267a3a414b64da393db4adc018f39d5, State: Initialized, Role: FOLLOWER
I20260620 18:37:30.058246 27040 raft_consensus.cc:348] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.058355 27039 consensus_queue.cc:260] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.058416 27040 raft_consensus.cc:374] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:30.058461 27040 raft_consensus.cc:729] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3709fb58fd604b2eaf7c8874db031a47, State: Initialized, Role: FOLLOWER
I20260620 18:37:30.058617 27039 ts_tablet_manager.cc:1428] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:37:30.058583 27040 consensus_queue.cc:260] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.058646 26897 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
I20260620 18:37:30.058872 27040 ts_tablet_manager.cc:1428] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:37:30.058939 27029 heartbeater.cc:503] Master 127.10.109.126:46501 was elected leader, sending a full tablet report...
W20260620 18:37:30.104876 26898 tablet.cc:2369] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:30.194895 26760 tablet.cc:2369] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:30.256312 27030 tablet.cc:2369] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:30.391450 27043 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:30.391606 27043 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.392056 27043 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613), 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315)
I20260620 18:37:30.395283 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.392286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33613 (local address 127.10.109.66:34977)
0620 18:37:30.392454 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.392465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.392549 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.392718 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:30.392719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:30.392721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:30.392800 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.392802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.393573 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.393574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.394367 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.394369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.395016 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.395017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.395090 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.395095 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:37:30.395150 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:37:30.395187 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260620 18:37:30.395308 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.392376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:34977 (local address 127.10.109.67:33613)
0620 18:37:30.392429 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.392431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.392595 (+ 164us) server_negotiation.cc:407] Connection header received
0620 18:37:30.392631 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.392632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.392644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.392704 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:30.392916 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.393476 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.394460 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.394934 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.394997 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.395005 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.395186 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.395227 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:37:30.395484 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.392404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:32923 (local address 127.10.109.65:40315)
0620 18:37:30.392527 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.392529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.392540 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:37:30.392691 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.392692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.392721 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.392801 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:30.393064 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.393590 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.394568 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.395083 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.395161 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.395224 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.395350 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.395399 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260620 18:37:30.395470 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.392274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.10.109.66:32923)
0620 18:37:30.392442 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.392455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.392567 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.392888 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:30.392890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:30.392893 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:30.392972 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.392973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.393678 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.393679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.394474 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:30.394476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.395189 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.395191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:30.395264 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.395268 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:37:30.395336 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:37:30.395377 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260620 18:37:30.395820 26713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a616d6838847f4a70428c340023ce0" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b0ba22bc4949f093afacba862a9105" is_pre_election: true
I20260620 18:37:30.395882 26982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a616d6838847f4a70428c340023ce0" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3709fb58fd604b2eaf7c8874db031a47" is_pre_election: true
I20260620 18:37:30.395975 26713 raft_consensus.cc:2454] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 0.
I20260620 18:37:30.396049 26982 raft_consensus.cc:2454] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 0.
I20260620 18:37:30.396214 26784 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105, a267a3a414b64da393db4adc018f39d5; no voters:
I20260620 18:37:30.396350 27043 raft_consensus.cc:2781] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:30.396406 27043 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:30.396433 27043 raft_consensus.cc:3037] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:30.397185 27043 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.397317 27043 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613), 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315)
I20260620 18:37:30.397428 26982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a616d6838847f4a70428c340023ce0" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3709fb58fd604b2eaf7c8874db031a47"
I20260620 18:37:30.397466 26713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a616d6838847f4a70428c340023ce0" candidate_uuid: "a267a3a414b64da393db4adc018f39d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b0ba22bc4949f093afacba862a9105"
I20260620 18:37:30.397506 26982 raft_consensus.cc:3037] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:30.397535 26713 raft_consensus.cc:3037] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:30.398303 26982 raft_consensus.cc:2454] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 1.
I20260620 18:37:30.398303 26713 raft_consensus.cc:2454] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a267a3a414b64da393db4adc018f39d5 in term 1.
I20260620 18:37:30.398476 26785 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5; no voters:
I20260620 18:37:30.398612 27043 raft_consensus.cc:2781] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:30.398772 27043 raft_consensus.cc:686] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 1 LEADER]: Becoming Leader. State: Replica: a267a3a414b64da393db4adc018f39d5, State: Running, Role: LEADER
I20260620 18:37:30.398862 27043 consensus_queue.cc:237] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:30.399744 26571 catalog_manager.cc:5495] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a267a3a414b64da393db4adc018f39d5 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "a267a3a414b64da393db4adc018f39d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } health_report { overall_health: UNKNOWN } } }
I20260620 18:37:30.489187 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:30.486647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.10.109.66:37147)
0620 18:37:30.486716 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:30.486718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:30.486729 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:37:30.486808 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:30.486809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:30.486837 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:30.486928 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:30.487071 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.487589 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.488364 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:30.488571 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:30.488650 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:30.488826 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:30.488933 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:30.489036 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:30.489061 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 18:37:30.489083 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:37:30.491159 26713 raft_consensus.cc:1261] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Refusing update from remote peer a267a3a414b64da393db4adc018f39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:30.491159 26982 raft_consensus.cc:1261] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Refusing update from remote peer a267a3a414b64da393db4adc018f39d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:30.491474 27043 consensus_queue.cc:1035] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:30.491674 27048 consensus_queue.cc:1035] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:30.493386 27051 mvcc.cc:205] Tried to move back new op lower bound from 7298992744408502272 to 7298992744034594816. Current Snapshot: MvccSnapshot[applied={T|T < 7298992744408502272}]
I20260620 18:37:30.494159 27053 mvcc.cc:205] Tried to move back new op lower bound from 7298992744408502272 to 7298992744034594816. Current Snapshot: MvccSnapshot[applied={T|T < 7298992744408502272}]
I20260620 18:37:30.496201 26570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:30.496300 26570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:30.497133 26570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Sending DeleteTablet for 3 replicas of tablet e9a616d6838847f4a70428c340023ce0
I20260620 18:37:30.497406 26831 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a616d6838847f4a70428c340023ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:30 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:52976
I20260620 18:37:30.497511 26963 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a616d6838847f4a70428c340023ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:30 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:58384
I20260620 18:37:30.497640 26681 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a616d6838847f4a70428c340023ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:30 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36882
I20260620 18:37:30.497720 27061 tablet_replica.cc:330] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: stopping tablet replica
I20260620 18:37:30.497814 27061 raft_consensus.cc:2229] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:30.497828 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26609
I20260620 18:37:30.497978 27061 raft_consensus.cc:2258] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:30.498696 27061 ts_tablet_manager.cc:1905] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:30.500231 27061 ts_tablet_manager.cc:1918] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:30.500314 27061 log.cc:1196] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/e9a616d6838847f4a70428c340023ce0
I20260620 18:37:30.500556 27061 ts_tablet_manager.cc:1939] T e9a616d6838847f4a70428c340023ce0 P 3709fb58fd604b2eaf7c8874db031a47: Deleting consensus metadata
I20260620 18:37:30.500927 26556 catalog_manager.cc:4841] TS 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): tablet e9a616d6838847f4a70428c340023ce0 (table test-table [id=6f9f6fc039fa44149f3271e78827a09c]) successfully deleted
I20260620 18:37:30.502857 27060 tablet_replica.cc:330] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: stopping tablet replica
I20260620 18:37:30.502972 27060 raft_consensus.cc:2229] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:30.503089 27060 raft_consensus.cc:2258] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:30.503463 27060 ts_tablet_manager.cc:1905] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:30.505002 27060 ts_tablet_manager.cc:1918] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:30.505152 27060 log.cc:1196] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/e9a616d6838847f4a70428c340023ce0
I20260620 18:37:30.505390 27060 ts_tablet_manager.cc:1939] T e9a616d6838847f4a70428c340023ce0 P a267a3a414b64da393db4adc018f39d5: Deleting consensus metadata
W20260620 18:37:30.505537 26555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:40315 (error 108)
I20260620 18:37:30.505875 26558 catalog_manager.cc:4841] TS a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): tablet e9a616d6838847f4a70428c340023ce0 (table test-table [id=6f9f6fc039fa44149f3271e78827a09c]) successfully deleted
W20260620 18:37:30.506189 27035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:30.505958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40315 (local address 127.0.0.1:41442)
0620 18:37:30.506002 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:30.506013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:30.506049 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:30.506151 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 18:37:30.506424 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26764
W20260620 18:37:30.506309 26555 catalog_manager.cc:4584] TS 51b0ba22bc4949f093afacba862a9105 (127.10.109.65:40315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9a616d6838847f4a70428c340023ce0: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:37:30.510880 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26900
I20260620 18:37:30.515096 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 26538
I20260620 18:37:50.520505 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--webserver_interface=127.10.109.126
--webserver_port=35989
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:50.599828 27064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.600009 27064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.600030 27064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.601161 27064 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:37:50.601198 27064 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:37:50.601212 27064 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:37:50.601222 27064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:50.601235 27064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.601248 27064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.601260 27064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:50.601272 27064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:50.602643 27064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:50.602888 27064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.603058 27064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:37:50.608235 27064 server_base.cc:1034] running on GCE node
W20260620 18:37:50.608156 27070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.608379 27072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.608309 27069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.608693 27064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.608981 27064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.610175 27064 hybrid_clock.cc:648] HybridClock initialized: now 1781980670610163 us; error 45 us; skew 500 ppm
Jun 20 18:37:50 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.612792 27064 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:37:50.613233 27064 webserver.cc:460] Webserver started at http://127.10.109.126:35989/ using document root <none> and password file <none>
I20260620 18:37:50.613481 27064 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.613528 27064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.614893 27064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:50.615612 27079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.615842 27064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:50.615942 27064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "2bdd574db784424d92f4796a2af2183b"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "06f5fac1990d0911dfecb724714d9252"
server_key_iv: "e4701b5eed05c4732b8db429e20795c6"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.616297 27064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.631942 27064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.634030 27064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.634145 27064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.638319 27064 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:46501
I20260620 18:37:50.638597 27131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:46501 every 8 connection(s)
I20260620 18:37:50.638737 27064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:50.640197 27132 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:37:50.640865 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap starting.
I20260620 18:37:50.645462 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27064
I20260620 18:37:50.645771 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:40315
--local_ip_for_outbound_sockets=127.10.109.65
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=34071
--webserver_interface=127.10.109.65
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:50.650492 27132 log.cc:824] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:50.652619 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:37:50.652984 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Bootstrap complete.
I20260620 18:37:50.655274 27132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:50.655524 27132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Initialized, Role: FOLLOWER
I20260620 18:37:50.655699 27132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:50.655776 27132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:50.655828 27132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:50.655879 27132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:37:50.656823 27132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:50.656952 27132 leader_election.cc:302] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bdd574db784424d92f4796a2af2183b; no voters:
I20260620 18:37:50.657151 27132 leader_election.cc:288] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:37:50.657186 27136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:37:50.657332 27136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [term 3 LEADER]: Becoming Leader. State: Replica: 2bdd574db784424d92f4796a2af2183b, State: Running, Role: LEADER
I20260620 18:37:50.657440 27132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:50.657428 27136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } }
I20260620 18:37:50.657693 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:50.657858 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:50.658030 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bdd574db784424d92f4796a2af2183b. Latest consensus state: current_term: 3 leader_uuid: "2bdd574db784424d92f4796a2af2183b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdd574db784424d92f4796a2af2183b" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46501 } } }
I20260620 18:37:50.658100 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:50.658474 27143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:50.659256 27143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:50.659406 27143 catalog_manager.cc:1259] Loaded cluster ID: 4eb4ec61df1946b38f45d4d1efe19050
I20260620 18:37:50.659672 27143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:50.660948 27143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:50.661171 27143 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Loaded TSK: 0
I20260620 18:37:50.661264 27143 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bdd574db784424d92f4796a2af2183b: Loaded TSK: 1
F20260620 18:37:50.661545 27143 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd233b059ad google::LogMessage::Fail() at ??:0
@ 0x7fd233b0888a google::LogMessage::SendToLog() at ??:0
@ 0x7fd233b054de google::LogMessage::Flush() at ??:0
@ 0x7fd233b06a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd236272104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd23626fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd23ba5f1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd23ba5f48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd23ba5f594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd23ba25c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd23ba2fad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2359d30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2359c8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd23682c6db start_thread at ??:0
@ 0x7fd23260361f clone at ??:0
W20260620 18:37:50.762418 27134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.762694 27134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.762732 27134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.764624 27134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.764693 27134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.764739 27134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:37:50.767083 27134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:40315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=34071
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:50.767328 27134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.767642 27134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:50.772998 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.773010 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.773010 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.773340 27134 server_base.cc:1034] running on GCE node
I20260620 18:37:50.773568 27134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.773821 27134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.774986 27134 hybrid_clock.cc:648] HybridClock initialized: now 1781980670774980 us; error 38 us; skew 500 ppm
Jun 20 18:37:50 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.777635 27134 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:37:50.778163 27134 webserver.cc:460] Webserver started at http://127.10.109.65:34071/ using document root <none> and password file <none>
I20260620 18:37:50.778429 27134 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.778503 27134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.779996 27134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:50.780745 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.780946 27134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:50.781040 27134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "51b0ba22bc4949f093afacba862a9105"
format_stamp: "Formatted at 2026-06-20 18:37:08 on dist-test-slave-3jql"
server_key: "3ce37b581fac504e7b965909f8e98a98"
server_key_iv: "fc916bf69f1edfc2196681a43bf7420b"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.781386 27134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.845981 27134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.849187 27134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.849411 27134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.849668 27134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:50.850133 27190 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 18:37:50.852102 27134 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:37:50.852188 27134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 18:37:50.852239 27134 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:37:50.852874 27134 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:37:50.852941 27134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 18:37:50.852986 27190 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Bootstrap starting.
I20260620 18:37:50.860033 27134 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:40315
I20260620 18:37:50.860076 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:40315 every 8 connection(s)
I20260620 18:37:50.860446 27134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:50.862973 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27134
I20260620 18:37:50.863247 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:37147
--local_ip_for_outbound_sockets=127.10.109.66
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=39169
--webserver_interface=127.10.109.66
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:50.863888 27190 log.cc:824] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:50.866639 27190 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:37:50.866995 27190 tablet_bootstrap.cc:492] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Bootstrap complete.
I20260620 18:37:50.868189 27190 ts_tablet_manager.cc:1397] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.014s
I20260620 18:37:50.869529 27190 raft_consensus.cc:348] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:50.869846 27190 raft_consensus.cc:729] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b0ba22bc4949f093afacba862a9105, State: Initialized, Role: FOLLOWER
I20260620 18:37:50.870008 27190 consensus_queue.cc:260] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:50.870260 27190 ts_tablet_manager.cc:1428] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260620 18:37:50.890945 27300 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:50.861099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:59423)
0620 18:37:50.861276 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.861318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.861759 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.890716 (+ 28957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
W20260620 18:37:50.891166 27298 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:46501 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:46501: Client connection negotiation failed: client connection to 127.10.109.126:46501: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:37:50.891502 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.891395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:35549)
0620 18:37:50.891450 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.891485 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:37:50.891834 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.891742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:52421)
0620 18:37:50.891788 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.891811 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:37:50.891944 27298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:37:50.950487 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.950711 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.950749 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.951887 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.951957 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.952004 27302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:37:50.953435 27302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=39169
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:50.953691 27302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.953917 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:37:50.958993 27302 server_base.cc:1034] running on GCE node
W20260620 18:37:50.958956 27311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.958956 27309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.958943 27308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.959307 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.959512 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.960678 27302 hybrid_clock.cc:648] HybridClock initialized: now 1781980670960657 us; error 36 us; skew 500 ppm
Jun 20 18:37:50 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.963176 27302 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:37:50.963663 27302 webserver.cc:460] Webserver started at http://127.10.109.66:39169/ using document root <none> and password file <none>
I20260620 18:37:50.963910 27302 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.963985 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.965346 27302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:50.965991 27318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.966162 27302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:37:50.966264 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "a267a3a414b64da393db4adc018f39d5"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "2c2c3fa08c633ceacb91942f7adad5ea"
server_key_iv: "8ac89c3ec8dabc632b71f0bd111a9925"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.966631 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.980000 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.981763 27302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.981932 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.982148 27302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:50.982447 27302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:50.982502 27302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.982542 27302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:50.982568 27302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.988735 27302 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:37147
I20260620 18:37:50.988771 27431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:37147 every 8 connection(s)
I20260620 18:37:50.989118 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:50.990085 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.989757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:60007)
0620 18:37:50.989897 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.989958 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
W20260620 18:37:50.990480 27432 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:46501 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:46501: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
I20260620 18:37:50.990993 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.990656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:45677)
0620 18:37:50.990881 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.990948 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 18:37:50.991611 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.991450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:42763)
0620 18:37:50.991541 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.991581 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 18:37:50.991825 27432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:37:50.998951 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27302
I20260620 18:37:50.999212 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:33613
--local_ip_for_outbound_sockets=127.10.109.67
--tserver_master_addrs=127.10.109.126:46501
--webserver_port=45347
--webserver_interface=127.10.109.67
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:51.077592 27435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:51.077780 27435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:51.077803 27435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:51.079077 27435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:51.079114 27435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:51.079141 27435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:37:51.080523 27435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:33613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=45347
--tserver_master_addrs=127.10.109.126:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:37:51.080749 27435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:51.080969 27435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:51.085868 27440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:51.085906 27441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:51.085994 27443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:51.086051 27435 server_base.cc:1034] running on GCE node
I20260620 18:37:51.086340 27435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:51.086562 27435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:51.087692 27435 hybrid_clock.cc:648] HybridClock initialized: now 1781980671087655 us; error 50 us; skew 500 ppm
Jun 20 18:37:51 dist-test-slave-3jql krb5kdc[25991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980671, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:51.090046 27435 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:37:51.090512 27435 webserver.cc:460] Webserver started at http://127.10.109.67:45347/ using document root <none> and password file <none>
I20260620 18:37:51.090776 27435 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:51.090849 27435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:51.092074 27435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:37:51.092638 27450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.092811 27435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:37:51.092877 27435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "3709fb58fd604b2eaf7c8874db031a47"
format_stamp: "Formatted at 2026-06-20 18:37:09 on dist-test-slave-3jql"
server_key: "4d3fa3496d6e4f1a2f6c1b826bf6d36a"
server_key_iv: "e0832b1c5311563bd496a6c9c3ccb30b"
server_key_version: "encryptionkey@0"
I20260620 18:37:51.093182 27435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:51.113288 27435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:51.115224 27435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:51.115368 27435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:51.115581 27435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:51.115864 27435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:51.115921 27435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.115962 27435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:51.115988 27435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.121673 27435 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:33613
I20260620 18:37:51.121712 27563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:33613 every 8 connection(s)
I20260620 18:37:51.122062 27435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:51.123090 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.122712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:59653)
0620 18:37:51.122900 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.122964 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
W20260620 18:37:51.123500 27564 heartbeater.cc:644] Failed to heartbeat to 127.10.109.126:46501 (0 consecutive failures): Network error: Failed to ping master at 127.10.109.126:46501: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
I20260620 18:37:51.123766 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.123673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:59705)
0620 18:37:51.123717 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.123742 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:37:51.123778 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27435
I20260620 18:37:51.124080 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.123973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:44641)
0620 18:37:51.124025 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.124052 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 18:37:51.124212 27564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:37:51.892546 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.892316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:53943)
0620 18:37:51.892452 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.892488 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260620 18:37:51.992365 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.992140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:35903)
0620 18:37:51.992278 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.992311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260620 18:37:52.124717 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.124509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:39797)
0620 18:37:52.124634 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:52.124668 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260620 18:37:52.296160 27571 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:52.296309 27571 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:52.296823 27571 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
Jun 20 18:37:52 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.66@KRBTEST.COM
Jun 20 18:37:52 dist-test-slave-3jql krb5kdc[25991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.67@KRBTEST.COM
I20260620 18:37:52.309090 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.297075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:40791 (local address 127.10.109.66:37147)
0620 18:37:52.297248 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:52.297261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:52.297295 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:37:52.297925 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:52.297927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:52.297978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:52.298149 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:52.299164 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.300559 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302099 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.302723 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.305504 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:52.305507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:52.305508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:52.305518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:52.307440 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.307887 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.307890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.307891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.307980 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.308230 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.308231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.308231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.308466 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:52.308561 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:52.308844 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:37:52.308964 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260620 18:37:52.309106 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.296959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:37147 (local address 127.10.109.65:40791)
0620 18:37:52.297019 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:52.297031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:52.297090 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:52.298239 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:52.298242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:52.298703 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:52.299013 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:52.299021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.300697 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.300698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:52.301995 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:52.301998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.302855 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:52.303005 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.303638 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:52.303648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:52.305378 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:52.307545 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:52.307552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:52.307554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:52.307775 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:52.308053 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:52.308054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:52.308054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:52.308160 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:52.308541 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:52.308548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:52.308687 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 18:37:52.308792 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:37:52.308976 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":93}
I20260620 18:37:52.309180 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.296981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:33613 (local address 127.10.109.65:33177)
0620 18:37:52.297110 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:52.297122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:52.297177 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:52.298239 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:52.298242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:52.298704 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:52.299013 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:52.299021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.300685 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.300687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:52.301995 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:52.301998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.302861 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:52.303005 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.303659 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:52.303663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:52.305822 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:52.307851 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:52.307853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:52.307853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:52.308038 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:52.308308 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:52.308309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:52.308309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:52.308375 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:52.308763 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:52.308764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:52.308809 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:37:52.308895 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:37:52.309042 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":87,"thread_start_us":52,"threads_started":1}
I20260620 18:37:52.309535 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.297103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:33177 (local address 127.10.109.67:33613)
0620 18:37:52.297248 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:52.297261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:52.297295 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:37:52.297915 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:52.297919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:52.297978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:52.298140 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:52.299163 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.300575 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302099 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.302723 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.302860 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.305931 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:52.305935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:52.305935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:52.305947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:52.307738 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.308134 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.308136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.308137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.308230 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.308444 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.308445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.308445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.308666 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:52.308753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:52.308956 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:37:52.309080 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
W20260620 18:37:52.310230 27187 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
W20260620 18:37:52.310303 27185 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
I20260620 18:37:52.310382 27185 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105; no voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5
I20260620 18:37:52.310530 27571 raft_consensus.cc:2726] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:52.893227 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.893011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:39769)
0620 18:37:52.893149 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:52.893183 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260620 18:37:52.993054 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.992849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:48661)
0620 18:37:52.992980 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:52.993016 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260620 18:37:53.125330 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:53.125127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:51927)
0620 18:37:53.125256 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:53.125293 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260620 18:37:53.893937 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:53.893687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:50197)
0620 18:37:53.893833 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:53.893869 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260620 18:37:53.993727 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:53.993533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:55705)
0620 18:37:53.993656 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:53.993692 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 18:37:54.125977 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:54.125772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:51235)
0620 18:37:54.125907 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:54.125942 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260620 18:37:54.482642 27581 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:54.482733 27581 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:54.483085 27581 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
W20260620 18:37:54.483480 27187 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
W20260620 18:37:54.483553 27185 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
I20260620 18:37:54.483613 27185 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105; no voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5
I20260620 18:37:54.483700 27581 raft_consensus.cc:2726] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:54.894603 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:54.894390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:36067)
0620 18:37:54.894525 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:54.894557 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 18:37:54.994341 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:54.994148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:58457)
0620 18:37:54.994276 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:54.994309 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260620 18:37:55.126606 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:55.126412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:49917)
0620 18:37:55.126539 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:55.126574 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260620 18:37:55.895345 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:55.895152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:58897)
0620 18:37:55.895268 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:55.895302 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 18:37:55.995093 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:55.994895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:45281)
0620 18:37:55.995020 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:55.995054 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260620 18:37:56.127211 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:56.127024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:44951)
0620 18:37:56.127141 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:56.127175 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260620 18:37:56.896112 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:56.895851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:59563)
0620 18:37:56.896009 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:56.896049 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260620 18:37:56.995754 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:56.995576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:42557)
0620 18:37:56.995684 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:56.995718 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260620 18:37:57.127879 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:57.127688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:51293)
0620 18:37:57.127809 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:57.127844 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260620 18:37:57.501377 27591 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:57.501494 27591 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:57.501718 27591 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
W20260620 18:37:57.502027 27187 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
W20260620 18:37:57.502112 27185 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
I20260620 18:37:57.502156 27185 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105; no voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5
I20260620 18:37:57.502234 27591 raft_consensus.cc:2726] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:57.897209 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:57.896986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:41643)
0620 18:37:57.897122 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:57.897159 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260620 18:37:57.996299 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:57.996116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:44315)
0620 18:37:57.996233 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:57.996266 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260620 18:37:58.128492 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.128261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:56035)
0620 18:37:58.128423 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:58.128457 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260620 18:37:58.897876 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.897650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:34315)
0620 18:37:58.897791 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:58.897830 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260620 18:37:58.996966 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.996740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:38145)
0620 18:37:58.996895 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:58.996929 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260620 18:37:59.129161 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.128952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:49661)
0620 18:37:59.129091 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:59.129125 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260620 18:37:59.756490 27598 raft_consensus.cc:482] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:59.756594 27598 raft_consensus.cc:504] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a267a3a414b64da393db4adc018f39d5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 37147 } } peers { permanent_uuid: "3709fb58fd604b2eaf7c8874db031a47" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 33613 } } peers { permanent_uuid: "51b0ba22bc4949f093afacba862a9105" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40315 } }
I20260620 18:37:59.756832 27598 leader_election.cc:288] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147), 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613)
W20260620 18:37:59.757452 27185 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3709fb58fd604b2eaf7c8874db031a47 (127.10.109.67:33613): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
W20260620 18:37:59.757539 27187 leader_election.cc:341] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a267a3a414b64da393db4adc018f39d5 (127.10.109.66:37147): Not found: Tablet not found: e9a616d6838847f4a70428c340023ce0
I20260620 18:37:59.757572 27187 leader_election.cc:302] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b0ba22bc4949f093afacba862a9105; no voters: 3709fb58fd604b2eaf7c8874db031a47, a267a3a414b64da393db4adc018f39d5
I20260620 18:37:59.757660 27598 raft_consensus.cc:2726] T e9a616d6838847f4a70428c340023ce0 P 51b0ba22bc4949f093afacba862a9105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:37:59.898727 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.898470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:60611)
0620 18:37:59.898635 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:59.898676 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260620 18:37:59.997691 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.997452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:38993)
0620 18:37:59.997599 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:59.997635 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260620 18:38:00.129846 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:00.129633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:38425)
0620 18:38:00.129757 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:00.129791 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260620 18:38:00.899456 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:00.899240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:47461)
0620 18:38:00.899369 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:00.899408 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260620 18:38:00.998605 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:00.998302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:59539)
0620 18:38:00.998496 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:00.998551 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260620 18:38:01.130983 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:01.130474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:54703)
0620 18:38:01.130694 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:01.130866 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260620 18:38:01.900383 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:01.900107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:54079)
0620 18:38:01.900279 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:01.900319 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260620 18:38:01.999571 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:01.999301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:42671)
0620 18:38:01.999471 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:01.999507 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260620 18:38:02.131773 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.131531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:58971)
0620 18:38:02.131695 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:02.131728 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260620 18:38:02.901407 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.901060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:52239)
0620 18:38:02.901293 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:02.901356 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
I20260620 18:38:03.000237 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.000015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:35315)
0620 18:38:03.000140 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.000194 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260620 18:38:03.132504 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.132209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:50229)
0620 18:38:03.132394 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.132450 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260620 18:38:03.902251 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.902002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:44291)
0620 18:38:03.902147 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.902195 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260620 18:38:04.001075 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:04.000821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:44531)
0620 18:38:04.000981 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:04.001027 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":59,"threads_started":1}
I20260620 18:38:04.133317 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:04.133085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:42169)
0620 18:38:04.133219 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:04.133267 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260620 18:38:04.903199 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:04.902981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:55611)
0620 18:38:04.903112 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:04.903148 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260620 18:38:05.001884 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:05.001648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:45895)
0620 18:38:05.001795 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:05.001833 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260620 18:38:05.134052 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:05.133841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.67:53581)
0620 18:38:05.133972 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:05.134007 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260620 18:38:05.903978 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:05.903758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.65:39293)
0620 18:38:05.903896 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:05.903932 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260620 18:38:06.002878 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.002451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:46501 (local address 127.10.109.66:45107)
0620 18:38:06.002657 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:06.002721 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.126:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260620 18:38:06.124588 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27134
I20260620 18:38:06.129390 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27302
I20260620 18:38:06.133700 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27435
2026-06-20T18:38:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58686 ms)
[----------] 30 tests from SecurityITest (233340 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:06 dist-test-slave-3jql krb5kdc[27621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:06 dist-test-slave-3jql krb5kdc[27621](info): set up 2 sockets
Jun 20 18:38:06 dist-test-slave-3jql krb5kdc[27621](info): commencing operation
krb5kdc: starting...
W20260620 18:38:07.207032 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:07.326061 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43763
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:07.406648 27634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:07.406921 27634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:07.406960 27634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:07.408123 27634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:07.408192 27634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:07.408223 27634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:07.408245 27634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:07.408272 27634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:07.409698 27634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:07.409960 27634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:07.410187 27634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:07.414892 27640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.414978 27634 server_base.cc:1034] running on GCE node
W20260620 18:38:07.414892 27642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.414925 27639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.415406 27634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:07.415694 27634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:07.416862 27634 hybrid_clock.cc:648] HybridClock initialized: now 1781980687416844 us; error 41 us; skew 500 ppm
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:07.419325 27634 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:07.419744 27634 webserver.cc:460] Webserver started at http://127.10.109.126:36183/ using document root <none> and password file <none>
I20260620 18:38:07.419916 27634 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:07.419953 27634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:07.420145 27634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:07.421162 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "36d93aa8ea844d309d3fa20f9a393670"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ecf79867721a713299039a09ee2cbb49"
server_key_iv: "a72679b8117619fd5769a70433027cec"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.421486 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "36d93aa8ea844d309d3fa20f9a393670"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ecf79867721a713299039a09ee2cbb49"
server_key_iv: "a72679b8117619fd5769a70433027cec"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.423220 27634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 18:38:07.424115 27649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.424369 27634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:07.424443 27634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "36d93aa8ea844d309d3fa20f9a393670"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ecf79867721a713299039a09ee2cbb49"
server_key_iv: "a72679b8117619fd5769a70433027cec"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.424494 27634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:07.446468 27634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:07.448823 27634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:07.448938 27634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:07.452622 27634 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:43763
I20260620 18:38:07.452687 27701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:43763 every 8 connection(s)
I20260620 18:38:07.453014 27634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:07.453924 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.456153 27702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: Bootstrap starting.
I20260620 18:38:07.456692 27702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.456931 27702 log.cc:824] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:07.457587 27702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: No bootstrap required, opened a new log
I20260620 18:38:07.459173 27702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } }
I20260620 18:38:07.459309 27702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.459331 27702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d93aa8ea844d309d3fa20f9a393670, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.459461 27702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } }
I20260620 18:38:07.459534 27702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:07.459556 27702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:07.459583 27702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.460047 27702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } }
I20260620 18:38:07.460126 27702 leader_election.cc:302] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d93aa8ea844d309d3fa20f9a393670; no voters:
I20260620 18:38:07.460246 27702 leader_election.cc:288] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:07.460350 27705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:07.460541 27705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [term 1 LEADER]: Becoming Leader. State: Replica: 36d93aa8ea844d309d3fa20f9a393670, State: Running, Role: LEADER
I20260620 18:38:07.460629 27705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } }
I20260620 18:38:07.460682 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27634
I20260620 18:38:07.460786 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:07.460979 10677 external_mini_cluster.cc:1428] Setting key c6ddb24d58305b18b329b023c4069163
I20260620 18:38:07.461031 27707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d93aa8ea844d309d3fa20f9a393670. Latest consensus state: current_term: 1 leader_uuid: "36d93aa8ea844d309d3fa20f9a393670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } } }
I20260620 18:38:07.461086 27702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:07.461108 27707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:07.461645 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d93aa8ea844d309d3fa20f9a393670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d93aa8ea844d309d3fa20f9a393670" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43763 } } }
I20260620 18:38:07.461715 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:07.461939 27709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:07.462463 27709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:07.464093 27709 catalog_manager.cc:1347] Generated new cluster ID: ed7fa1c122e14800851e90f11e486355
I20260620 18:38:07.464151 27709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:07.472718 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.461571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.10.109.126:43763)
0620 18:38:07.461743 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.461754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.461796 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:38:07.461920 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.461927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.461990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.462148 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.462929 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.464021 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.465112 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.465744 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.465856 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.468599 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.468602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.468603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.468613 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.471108 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.471512 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.471515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.471515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.471620 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.471836 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.471837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.471837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.472117 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.472271 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.472380 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.472501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 18:38:07.491220 27709 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:07.491825 27709 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:07.497303 27709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: Generated new TSK 0
I20260620 18:38:07.497658 27709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:38:07.538444 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:07.621145 27728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:07.621377 27728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:07.621418 27728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:07.622536 27728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:07.622603 27728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:07.622648 27728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:07.624100 27728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:07.624372 27728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:07.624590 27728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:07.629756 27734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.629734 27733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.629844 27728 server_base.cc:1034] running on GCE node
W20260620 18:38:07.629737 27736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.630157 27728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:07.630369 27728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:07.631567 27728 hybrid_clock.cc:648] HybridClock initialized: now 1781980687631549 us; error 29 us; skew 500 ppm
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:07.634001 27728 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:38:07.634374 27728 webserver.cc:460] Webserver started at http://127.10.109.65:45849/ using document root <none> and password file <none>
I20260620 18:38:07.634545 27728 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:07.634583 27728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:07.634668 27728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:07.635787 27728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "66f8a51def0449b3a27a26582c6f73f3"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "a11637071ea292a7081aa3f87af3cb76"
server_key_iv: "8053f12310072ceafbc283458eb437ce"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.636081 27728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "66f8a51def0449b3a27a26582c6f73f3"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "a11637071ea292a7081aa3f87af3cb76"
server_key_iv: "8053f12310072ceafbc283458eb437ce"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.637312 27728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:38:07.637991 27743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.638216 27728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:07.638335 27728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "66f8a51def0449b3a27a26582c6f73f3"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "a11637071ea292a7081aa3f87af3cb76"
server_key_iv: "8053f12310072ceafbc283458eb437ce"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.638414 27728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:07.649001 27728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:07.651074 27728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:07.651237 27728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:07.651468 27728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:07.651786 27728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:07.651823 27728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.651846 27728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:07.651860 27728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.657900 27728 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:46753
I20260620 18:38:07.657974 27856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:46753 every 8 connection(s)
I20260620 18:38:07.658308 27728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:07.663903 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27728
I20260620 18:38:07.664002 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:07.664222 10677 external_mini_cluster.cc:1428] Setting key 8b3c1d2d3488b88d223089d250d9e15c
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:07.668905 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.659088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:34019 (local address 127.10.109.126:43763)
0620 18:38:07.659166 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.659168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.659381 (+ 213us) server_negotiation.cc:407] Connection header received
0620 18:38:07.660339 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.660340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.660351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.660409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.661173 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.661830 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.662971 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.663178 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.663272 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.665575 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.665576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.665577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.665583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.667390 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.667921 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.667922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.667922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.667976 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.668284 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.668284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.668285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.668386 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.668441 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.668769 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.668835 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:38:07.669243 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.659069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43763 (local address 127.10.109.65:34019)
0620 18:38:07.659274 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.659310 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.660236 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.660467 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.660470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.660727 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.660923 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.660927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.661923 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.661924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.662870 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.662873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.663258 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.663259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.663353 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.663875 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.663883 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.665466 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.667477 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.667484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.667487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.667837 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.668047 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.668048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.668048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.668195 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.668456 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.668462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.668623 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.668746 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.668918 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260620 18:38:07.669869 27857 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43763
I20260620 18:38:07.669976 27857 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:07.670181 27857 heartbeater.cc:511] Master 127.10.109.126:43763 requested a full tablet report, sending...
I20260620 18:38:07.670801 27666 ts_manager.cc:194] Registered new tserver with Master: 66f8a51def0449b3a27a26582c6f73f3 (127.10.109.65:46753)
I20260620 18:38:07.671765 27666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:34019
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:38:07.689481 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:07.770226 27862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:07.770414 27862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:07.770437 27862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:07.771602 27862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:07.771646 27862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:07.771670 27862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:07.773087 27862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:07.773321 27862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:07.773528 27862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:07.778656 27867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.778661 27870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.778666 27868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.779036 27862 server_base.cc:1034] running on GCE node
I20260620 18:38:07.779250 27862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:07.779500 27862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:07.780689 27862 hybrid_clock.cc:648] HybridClock initialized: now 1781980687780683 us; error 37 us; skew 500 ppm
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:07.783252 27862 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:38:07.783707 27862 webserver.cc:460] Webserver started at http://127.10.109.66:46711/ using document root <none> and password file <none>
I20260620 18:38:07.783963 27862 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:07.784034 27862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:07.784183 27862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:07.785246 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "dc3d7523913a442c814050069a9ad452"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ce27905fd8cd8a70231f18530f1ccafe"
server_key_iv: "7769a9b0ead2a937e017ae8bc1403532"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.785607 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "dc3d7523913a442c814050069a9ad452"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ce27905fd8cd8a70231f18530f1ccafe"
server_key_iv: "7769a9b0ead2a937e017ae8bc1403532"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.787012 27862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:07.787784 27877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.787964 27862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:07.788066 27862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "dc3d7523913a442c814050069a9ad452"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "ce27905fd8cd8a70231f18530f1ccafe"
server_key_iv: "7769a9b0ead2a937e017ae8bc1403532"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.788163 27862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:07.800249 27862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:07.802408 27862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:07.802579 27862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:07.802872 27862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:07.803193 27862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:07.803252 27862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.803292 27862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:07.803320 27862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.809798 27862 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:35567
I20260620 18:38:07.809834 27990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:35567 every 8 connection(s)
I20260620 18:38:07.810204 27862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:07.813964 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27862
I20260620 18:38:07.814106 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:07.814309 10677 external_mini_cluster.cc:1428] Setting key e40dba75f2e7a05a093532792536e0d4
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:07.819936 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.810988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:52507 (local address 127.10.109.126:43763)
0620 18:38:07.811038 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.811040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.811258 (+ 218us) server_negotiation.cc:407] Connection header received
0620 18:38:07.811737 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.811738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.811751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.811804 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.812751 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.813253 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.814304 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.814598 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.814682 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.816832 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.816833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.816834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.816841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.818420 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.818881 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.818883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.818883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.818946 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.819181 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.819182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.819182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.819329 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.819406 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.819703 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.819798 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:38:07.819954 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.810935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43763 (local address 127.10.109.66:52507)
0620 18:38:07.811094 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.811139 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.811615 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.811871 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.811875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.812451 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.812648 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.812653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.813349 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.813350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.814200 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.814203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.814687 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.814688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.814851 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.815399 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.815407 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.816744 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.818505 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.818511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.818513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.818796 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.819018 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.819018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.819018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.819114 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.819418 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.819425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.819555 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.819652 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.819797 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260620 18:38:07.820461 27991 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43763
I20260620 18:38:07.820580 27991 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:07.820780 27991 heartbeater.cc:511] Master 127.10.109.126:43763 requested a full tablet report, sending...
I20260620 18:38:07.821223 27666 ts_manager.cc:194] Registered new tserver with Master: dc3d7523913a442c814050069a9ad452 (127.10.109.66:35567)
I20260620 18:38:07.821754 27666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:52507
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:38:07.841564 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:07.925426 27996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:07.925637 27996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:07.925660 27996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:07.927103 27996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:07.927156 27996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:07.927186 27996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:07.928761 27996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:07.928946 27996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:07.929119 27996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:07.936203 28004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.936101 28002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:07.936096 28001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:07.936177 27996 server_base.cc:1034] running on GCE node
I20260620 18:38:07.936688 27996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:07.936945 27996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:07.938225 27996 hybrid_clock.cc:648] HybridClock initialized: now 1781980687938214 us; error 56 us; skew 500 ppm
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:07.941983 27996 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:38:07.942396 27996 webserver.cc:460] Webserver started at http://127.10.109.67:37765/ using document root <none> and password file <none>
I20260620 18:38:07.942631 27996 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:07.942688 27996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:07.942863 27996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:07.943958 27996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "361f5978a9b24bc0a7b3209e6471ba61"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "59c054767926de994cc732901a83fa01"
server_key_iv: "9470262e4929d12444beb7d401c3afc6"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.944343 27996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "361f5978a9b24bc0a7b3209e6471ba61"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "59c054767926de994cc732901a83fa01"
server_key_iv: "9470262e4929d12444beb7d401c3afc6"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.945904 27996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:07.946851 28011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.947036 27996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:07.947113 27996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "361f5978a9b24bc0a7b3209e6471ba61"
format_stamp: "Formatted at 2026-06-20 18:38:07 on dist-test-slave-3jql"
server_key: "59c054767926de994cc732901a83fa01"
server_key_iv: "9470262e4929d12444beb7d401c3afc6"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.947173 27996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:07.964056 27996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:07.966329 27996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:07.966444 27996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:07.966624 27996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:07.966944 27996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:07.966979 27996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.967000 27996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:07.967013 27996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.972811 27996 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:40601
I20260620 18:38:07.972860 28124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:40601 every 8 connection(s)
I20260620 18:38:07.973155 27996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:07.976816 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 27996
I20260620 18:38:07.976894 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:07.977075 10677 external_mini_cluster.cc:1428] Setting key 73ea7e5c530cf4b366ed18ba30a9d02b
Jun 20 18:38:07 dist-test-slave-3jql krb5kdc[27621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:07.983583 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.974065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:60059 (local address 127.10.109.126:43763)
0620 18:38:07.974159 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.974161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.974354 (+ 193us) server_negotiation.cc:407] Connection header received
0620 18:38:07.974967 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.974968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.974981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.975045 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.976030 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.976540 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.977805 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.977994 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.978079 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.980561 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.980562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.980563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.980569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.982056 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.982501 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.982502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.982502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.982563 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.982848 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.982849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.982849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.982962 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.983039 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.983409 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.983491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:38:07.983812 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.974019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43763 (local address 127.10.109.67:60059)
0620 18:38:07.974214 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.974253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.974859 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.975118 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.975122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.975705 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.975935 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.975940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.976635 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.976636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.977668 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.977670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.978089 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.978090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.978220 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.978980 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.978988 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.980444 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.982158 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.982164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.982166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.982419 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.982631 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.982632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.982633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.982782 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.983029 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.983035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.983173 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.983371 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.983541 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260620 18:38:07.984385 28125 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43763
I20260620 18:38:07.984476 28125 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:07.984656 28125 heartbeater.cc:511] Master 127.10.109.126:43763 requested a full tablet report, sending...
I20260620 18:38:07.985316 27666 ts_manager.cc:194] Registered new tserver with Master: 361f5978a9b24bc0a7b3209e6471ba61 (127.10.109.67:40601)
I20260620 18:38:07.985698 27666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:60059
I20260620 18:38:07.989655 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:07.998248 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.991246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.10.109.126:43763)
0620 18:38:07.991315 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.991317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.991343 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:38:07.991443 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.991443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.991461 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.991523 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.992097 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.992583 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.993365 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.993549 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.993659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.995059 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.995061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.995061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.995068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.996505 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.997067 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.997069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.997070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.997161 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.997572 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.997574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.997574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.997768 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.997888 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.998042 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.998137 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:38:08.000357 27666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:08.001953 27666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:08.008850 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35567 (local address 127.0.0.1:43040)
0620 18:38:08.004590 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.004619 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.004704 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.005172 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.005175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.005178 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.005309 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.005312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.006415 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.006416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.007645 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.007648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.008550 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.008630 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.008643 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.008710 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.008771 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 18:38:08.008863 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.10.109.66:35567)
0620 18:38:08.005012 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.005016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.005025 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:08.005056 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.005058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.005070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.005167 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.005400 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.006309 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.007735 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.008454 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008547 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.008608 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.008739 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.008782 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":81,"threads_started":1}
I20260620 18:38:08.009385 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:46753 (local address 127.0.0.1:55988)
0620 18:38:08.005407 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.005416 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.005459 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.005748 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.005749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.005752 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.005798 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.005799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.007067 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.007068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.008240 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.008243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.009127 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.009128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.009194 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.009198 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.009259 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.009301 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":70,"threads_started":1}
I20260620 18:38:08.009405 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.10.109.67:40601)
0620 18:38:08.005337 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.005341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.005385 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:38:08.005430 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.005433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.005449 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.005522 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.005692 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.006804 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008039 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.008655 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008736 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.008784 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.009274 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.009320 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":86,"threads_started":1}
I20260620 18:38:08.009682 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.10.109.65:46753)
0620 18:38:08.005128 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.005132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.005520 (+ 388us) server_negotiation.cc:407] Connection header received
0620 18:38:08.005570 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.005572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.005587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.005653 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.005883 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.006962 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008330 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.009061 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.009124 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.009170 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.009562 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.009607 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260620 18:38:08.009843 27925 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6da99c60a44064a0f6c475af86af0e (DEFAULT_TABLE table=test-table [id=1b2d04c2b4894717a803cdc973ea75e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.010246 27925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6da99c60a44064a0f6c475af86af0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.010254 27791 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6da99c60a44064a0f6c475af86af0e (DEFAULT_TABLE table=test-table [id=1b2d04c2b4894717a803cdc973ea75e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.010576 27791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6da99c60a44064a0f6c475af86af0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.010910 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.004659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:40601 (local address 127.0.0.1:57306)
0620 18:38:08.005278 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.005290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.005337 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.005585 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.005586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.005588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.005635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.005637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.006906 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.006907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.007930 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.007932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.008820 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.008821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.008884 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.008888 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.009235 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.009287 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":66,"threads_started":1}
I20260620 18:38:08.013324 28142 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: Bootstrap starting.
I20260620 18:38:08.013351 28143 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Bootstrap starting.
I20260620 18:38:08.014014 28142 tablet_bootstrap.cc:654] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.014058 28143 tablet_bootstrap.cc:654] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.014187 28059 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6da99c60a44064a0f6c475af86af0e (DEFAULT_TABLE table=test-table [id=1b2d04c2b4894717a803cdc973ea75e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.014564 28142 log.cc:824] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.014688 28059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6da99c60a44064a0f6c475af86af0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.014786 28143 log.cc:824] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.015377 28142 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: No bootstrap required, opened a new log
I20260620 18:38:08.015686 28142 ts_tablet_manager.cc:1397] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:08.017385 28146 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Bootstrap starting.
I20260620 18:38:08.017362 28142 raft_consensus.cc:348] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.017551 28142 raft_consensus.cc:374] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.017588 28142 raft_consensus.cc:729] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f8a51def0449b3a27a26582c6f73f3, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.017725 28142 consensus_queue.cc:260] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.018006 28142 ts_tablet_manager.cc:1428] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:38:08.018038 28146 tablet_bootstrap.cc:654] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.018055 27857 heartbeater.cc:503] Master 127.10.109.126:43763 was elected leader, sending a full tablet report...
I20260620 18:38:08.018656 28143 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: No bootstrap required, opened a new log
I20260620 18:38:08.018656 28146 log.cc:824] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.018726 28143 ts_tablet_manager.cc:1397] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 18:38:08.019485 28146 tablet_bootstrap.cc:492] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: No bootstrap required, opened a new log
I20260620 18:38:08.019563 28146 ts_tablet_manager.cc:1397] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:08.020838 28143 raft_consensus.cc:348] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.021019 28143 raft_consensus.cc:374] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.021054 28143 raft_consensus.cc:729] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3d7523913a442c814050069a9ad452, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.021165 28143 consensus_queue.cc:260] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.021461 28143 ts_tablet_manager.cc:1428] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:08.021490 27991 heartbeater.cc:503] Master 127.10.109.126:43763 was elected leader, sending a full tablet report...
I20260620 18:38:08.021507 28146 raft_consensus.cc:348] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.021661 28146 raft_consensus.cc:374] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.021685 28146 raft_consensus.cc:729] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361f5978a9b24bc0a7b3209e6471ba61, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.021826 28146 consensus_queue.cc:260] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.022101 28125 heartbeater.cc:503] Master 127.10.109.126:43763 was elected leader, sending a full tablet report...
I20260620 18:38:08.022294 28146 ts_tablet_manager.cc:1428] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 18:38:08.060770 27992 tablet.cc:2369] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:08.061793 28149 raft_consensus.cc:482] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:08.061888 28149 raft_consensus.cc:504] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.062206 28149 leader_election.cc:288] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 361f5978a9b24bc0a7b3209e6471ba61 (127.10.109.67:40601), 66f8a51def0449b3a27a26582c6f73f3 (127.10.109.65:46753)
I20260620 18:38:08.065163 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.062483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:34563 (local address 127.10.109.67:40601)
0620 18:38:08.062544 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.062546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.062557 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:08.062638 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.062639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.062649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.062695 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.062964 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.063458 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064387 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.064777 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.064904 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.065064 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.065102 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:38:08.065163 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.062402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:40601 (local address 127.10.109.66:34563)
0620 18:38:08.062475 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.062490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.062554 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.062772 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.062773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.062774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.062880 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.062883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.063554 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.063556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.064278 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.064279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.064888 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.064977 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.064980 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.065046 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.065088 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260620 18:38:08.065258 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.062593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:36337 (local address 127.10.109.65:46753)
0620 18:38:08.062641 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.062643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.062820 (+ 177us) server_negotiation.cc:407] Connection header received
0620 18:38:08.062888 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.062889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.062899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.062946 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.063100 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.063556 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064509 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.064910 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064994 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.065020 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.065162 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.065196 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:38:08.065269 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.062524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:46753 (local address 127.10.109.66:36337)
0620 18:38:08.062719 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.062730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.062830 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.062951 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.062952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.062954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.063028 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.063029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.063641 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.063643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.064394 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.064396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.064994 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.064995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.065069 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.065073 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.065145 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.065185 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":124,"threads_started":1}
I20260620 18:38:08.065565 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6da99c60a44064a0f6c475af86af0e" candidate_uuid: "dc3d7523913a442c814050069a9ad452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f8a51def0449b3a27a26582c6f73f3" is_pre_election: true
I20260620 18:38:08.065685 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6da99c60a44064a0f6c475af86af0e" candidate_uuid: "dc3d7523913a442c814050069a9ad452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361f5978a9b24bc0a7b3209e6471ba61" is_pre_election: true
I20260620 18:38:08.065730 27811 raft_consensus.cc:2454] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3d7523913a442c814050069a9ad452 in term 0.
I20260620 18:38:08.065814 28079 raft_consensus.cc:2454] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3d7523913a442c814050069a9ad452 in term 0.
I20260620 18:38:08.065948 27881 leader_election.cc:302] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f8a51def0449b3a27a26582c6f73f3, dc3d7523913a442c814050069a9ad452; no voters:
I20260620 18:38:08.066118 28149 raft_consensus.cc:2781] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:08.066187 28149 raft_consensus.cc:482] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:08.066231 28149 raft_consensus.cc:3037] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.067028 28149 raft_consensus.cc:504] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.067162 28149 leader_election.cc:288] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [CANDIDATE]: Term 1 election: Requested vote from peers 361f5978a9b24bc0a7b3209e6471ba61 (127.10.109.67:40601), 66f8a51def0449b3a27a26582c6f73f3 (127.10.109.65:46753)
I20260620 18:38:08.067613 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6da99c60a44064a0f6c475af86af0e" candidate_uuid: "dc3d7523913a442c814050069a9ad452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361f5978a9b24bc0a7b3209e6471ba61"
I20260620 18:38:08.067678 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6da99c60a44064a0f6c475af86af0e" candidate_uuid: "dc3d7523913a442c814050069a9ad452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f8a51def0449b3a27a26582c6f73f3"
I20260620 18:38:08.067721 28079 raft_consensus.cc:3037] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.067785 27811 raft_consensus.cc:3037] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.068447 28079 raft_consensus.cc:2454] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3d7523913a442c814050069a9ad452 in term 1.
I20260620 18:38:08.068626 27811 raft_consensus.cc:2454] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3d7523913a442c814050069a9ad452 in term 1.
I20260620 18:38:08.068691 27881 leader_election.cc:302] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361f5978a9b24bc0a7b3209e6471ba61, dc3d7523913a442c814050069a9ad452; no voters:
I20260620 18:38:08.068831 28149 raft_consensus.cc:2781] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:08.068984 28149 raft_consensus.cc:686] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 1 LEADER]: Becoming Leader. State: Replica: dc3d7523913a442c814050069a9ad452, State: Running, Role: LEADER
I20260620 18:38:08.069080 28149 consensus_queue.cc:237] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } }
I20260620 18:38:08.069916 27666 catalog_manager.cc:5495] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc3d7523913a442c814050069a9ad452 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "dc3d7523913a442c814050069a9ad452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3d7523913a442c814050069a9ad452" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:08.082355 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.080057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.10.109.66:35567)
0620 18:38:08.080106 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.080108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.080117 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:08.080169 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.080170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.080187 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.080270 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:08.080383 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.080830 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.081593 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.081785 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.081876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.082042 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:08.082136 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:08.082229 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.082249 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.082269 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:38:08.083972 28079 raft_consensus.cc:1261] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 1 FOLLOWER]: Refusing update from remote peer dc3d7523913a442c814050069a9ad452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:08.083977 27811 raft_consensus.cc:1261] T 0e6da99c60a44064a0f6c475af86af0e P 66f8a51def0449b3a27a26582c6f73f3 [term 1 FOLLOWER]: Refusing update from remote peer dc3d7523913a442c814050069a9ad452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:08.084342 28152 consensus_queue.cc:1035] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f8a51def0449b3a27a26582c6f73f3" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:08.084461 28149 consensus_queue.cc:1035] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "361f5978a9b24bc0a7b3209e6471ba61" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:08.086722 28155 mvcc.cc:205] Tried to move back new op lower bound from 7298992898389463040 to 7298992898331578368. Current Snapshot: MvccSnapshot[applied={T|T < 7298992898389463040}]
I20260620 18:38:08.086839 28157 mvcc.cc:205] Tried to move back new op lower bound from 7298992898389463040 to 7298992898331578368. Current Snapshot: MvccSnapshot[applied={T|T < 7298992898389463040}]
I20260620 18:38:08.087345 28156 mvcc.cc:205] Tried to move back new op lower bound from 7298992898389463040 to 7298992898331578368. Current Snapshot: MvccSnapshot[applied={T|T < 7298992898389463040 or (T in {7298992898389463040})}]
I20260620 18:38:08.089155 27666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:08.089246 27666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:08.089946 27666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d93aa8ea844d309d3fa20f9a393670: Sending DeleteTablet for 3 replicas of tablet 0e6da99c60a44064a0f6c475af86af0e
I20260620 18:38:08.090348 27791 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6da99c60a44064a0f6c475af86af0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:55988
I20260620 18:38:08.090374 28059 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6da99c60a44064a0f6c475af86af0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:57306
I20260620 18:38:08.090492 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27728
I20260620 18:38:08.091763 27925 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6da99c60a44064a0f6c475af86af0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:43040
I20260620 18:38:08.091791 28166 tablet_replica.cc:330] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: stopping tablet replica
I20260620 18:38:08.091892 28166 raft_consensus.cc:2229] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:08.091990 28166 raft_consensus.cc:2258] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:08.092151 28167 tablet_replica.cc:330] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: stopping tablet replica
I20260620 18:38:08.092283 28167 raft_consensus.cc:2229] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:08.092401 28167 raft_consensus.cc:2258] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:08.092844 28167 ts_tablet_manager.cc:1905] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:08.093184 28166 ts_tablet_manager.cc:1905] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:08.094959 28167 ts_tablet_manager.cc:1918] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:08.095091 28167 log.cc:1196] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/0e6da99c60a44064a0f6c475af86af0e
I20260620 18:38:08.095443 28167 ts_tablet_manager.cc:1939] T 0e6da99c60a44064a0f6c475af86af0e P dc3d7523913a442c814050069a9ad452: Deleting consensus metadata
I20260620 18:38:08.095885 27653 catalog_manager.cc:4841] TS dc3d7523913a442c814050069a9ad452 (127.10.109.66:35567): tablet 0e6da99c60a44064a0f6c475af86af0e (table test-table [id=1b2d04c2b4894717a803cdc973ea75e8]) successfully deleted
I20260620 18:38:08.096753 28166 ts_tablet_manager.cc:1918] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:08.096825 28166 log.cc:1196] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/0e6da99c60a44064a0f6c475af86af0e
I20260620 18:38:08.097091 28166 ts_tablet_manager.cc:1939] T 0e6da99c60a44064a0f6c475af86af0e P 361f5978a9b24bc0a7b3209e6471ba61: Deleting consensus metadata
W20260620 18:38:08.097347 27653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:46753 (error 108)
I20260620 18:38:08.097581 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27862
I20260620 18:38:08.098271 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.097726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:46753 (local address 127.0.0.1:56004)
0620 18:38:08.098193 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.098232 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
W20260620 18:38:08.102224 27653 catalog_manager.cc:4584] TS 66f8a51def0449b3a27a26582c6f73f3 (127.10.109.65:46753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e6da99c60a44064a0f6c475af86af0e: Network error: Client connection negotiation failed: client connection to 127.10.109.65:46753: connect: Connection refused (error 111)
I20260620 18:38:08.102478 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27996
I20260620 18:38:08.107264 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 27634
I20260620 18:38:08.107300 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.107142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:40601 (local address 127.0.0.1:57312)
0620 18:38:08.107232 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.107267 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.67:40601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
2026-06-20T18:38:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1973 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:08Z Disabled control of system clock
I20260620 18:38:08.135056 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43469
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:39173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:08.215086 28172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:08.215301 28172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:08.215320 28172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:08.216369 28172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:08.216408 28172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:08.216420 28172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:08.216432 28172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:08.216444 28172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:08.217667 28172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:43469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:08.217855 28172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:08.218067 28172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:08.222975 28178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.223011 28172 server_base.cc:1034] running on GCE node
W20260620 18:38:08.222983 28180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.222983 28177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.223443 28172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:08.223758 28172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:08.224948 28172 hybrid_clock.cc:648] HybridClock initialized: now 1781980688224928 us; error 38 us; skew 500 ppm
I20260620 18:38:08.226190 28172 webserver.cc:460] Webserver started at http://127.10.109.126:46499/ using document root <none> and password file <none>
I20260620 18:38:08.226481 28172 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:08.226557 28172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:08.226697 28172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:08.227885 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "5cf6e9105c09427bb1e47a38314bfd70"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "0938b84ad7f8aa30300a122db99f505c"
server_key_iv: "1af066833cc458bbf9c94aa5a4744805"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.228267 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "5cf6e9105c09427bb1e47a38314bfd70"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "0938b84ad7f8aa30300a122db99f505c"
server_key_iv: "1af066833cc458bbf9c94aa5a4744805"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.229580 28172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:08.230530 28186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.230863 28172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:08.230968 28172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "5cf6e9105c09427bb1e47a38314bfd70"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "0938b84ad7f8aa30300a122db99f505c"
server_key_iv: "1af066833cc458bbf9c94aa5a4744805"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.231051 28172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:08.249456 28172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:08.249780 28172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:08.249938 28172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:08.254415 28172 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:43469
I20260620 18:38:08.254455 28238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:43469 every 8 connection(s)
I20260620 18:38:08.254842 28172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:08.255918 28239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.258143 28239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: Bootstrap starting.
I20260620 18:38:08.258638 28239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.258968 28239 log.cc:824] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.259419 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28172
I20260620 18:38:08.259502 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:08.259698 10677 external_mini_cluster.cc:1428] Setting key 23129260fdd2801a1a20380793b57a76
I20260620 18:38:08.259804 28239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: No bootstrap required, opened a new log
I20260620 18:38:08.262379 28239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } }
I20260620 18:38:08.262552 28239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.262589 28239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf6e9105c09427bb1e47a38314bfd70, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.262712 28239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } }
I20260620 18:38:08.262823 28239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:08.262856 28239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:08.262894 28239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.263558 28239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } }
I20260620 18:38:08.263667 28239 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf6e9105c09427bb1e47a38314bfd70; no voters:
I20260620 18:38:08.263867 28239 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:08.263895 28244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:08.264017 28244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf6e9105c09427bb1e47a38314bfd70, State: Running, Role: LEADER
I20260620 18:38:08.264118 28239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:08.264139 28244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } }
I20260620 18:38:08.264508 28244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf6e9105c09427bb1e47a38314bfd70. Latest consensus state: current_term: 1 leader_uuid: "5cf6e9105c09427bb1e47a38314bfd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } } }
I20260620 18:38:08.264611 28244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:08.264582 28246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf6e9105c09427bb1e47a38314bfd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf6e9105c09427bb1e47a38314bfd70" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 43469 } } }
I20260620 18:38:08.264813 28246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:08.265128 28250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:08.265810 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.260202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.10.109.126:43469)
0620 18:38:08.260408 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.260415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.260439 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:08.261093 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.261100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.261160 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.261299 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.261511 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.262418 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.263547 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.264178 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.264343 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.265013 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.265018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.265019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:08.265229 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.265275 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:08.265290 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.265358 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.265484 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.265561 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260620 18:38:08.265635 28250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:08.267452 28250 catalog_manager.cc:1347] Generated new cluster ID: 41fc0567cd1b4297b8c7bd00f5e0ac24
I20260620 18:38:08.267503 28250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:08.277633 28250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:08.278110 28250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:08.281262 28250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: Generated new TSK 0
I20260620 18:38:08.281425 28250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:08.286666 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--builtin_ntp_servers=127.10.109.84:39173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:08.370877 28263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:08.371093 28263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:08.371130 28263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:08.372274 28263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:08.372319 28263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:08.372344 28263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:08.373685 28263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:08.373930 28263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:08.374226 28263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:08.379926 28271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.379925 28269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.380158 28263 server_base.cc:1034] running on GCE node
W20260620 18:38:08.379956 28268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.380483 28263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:08.380751 28263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:08.381920 28263 hybrid_clock.cc:648] HybridClock initialized: now 1781980688381895 us; error 40 us; skew 500 ppm
I20260620 18:38:08.383188 28263 webserver.cc:460] Webserver started at http://127.10.109.65:45729/ using document root <none> and password file <none>
I20260620 18:38:08.383425 28263 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:08.383494 28263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:08.383656 28263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:08.384683 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "4b3857b4fd504c05854a06c964f66349"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "9e426a39c64c96b3f2f149e14727694a"
server_key_iv: "4f8075d37116130afd0ffc2e0b971651"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.385083 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3857b4fd504c05854a06c964f66349"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "9e426a39c64c96b3f2f149e14727694a"
server_key_iv: "4f8075d37116130afd0ffc2e0b971651"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.386433 28263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:08.387360 28277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.387557 28263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 18:38:08.387681 28263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "4b3857b4fd504c05854a06c964f66349"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "9e426a39c64c96b3f2f149e14727694a"
server_key_iv: "4f8075d37116130afd0ffc2e0b971651"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.387769 28263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:08.403559 28263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:08.403899 28263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:08.404060 28263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:08.404294 28263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:08.404618 28263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:08.404673 28263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.404708 28263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:08.404759 28263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.410454 28263 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:42697
I20260620 18:38:08.410528 28390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:42697 every 8 connection(s)
I20260620 18:38:08.410926 28263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:08.411882 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28263
I20260620 18:38:08.412034 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:08.412283 10677 external_mini_cluster.cc:1428] Setting key b4684013ec66bc99d8db63cb6d0d4360
I20260620 18:38:08.413348 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--builtin_ntp_servers=127.10.109.84:39173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:38:08.417388 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.411704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:52031 (local address 127.10.109.126:43469)
0620 18:38:08.411778 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.411781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.412097 (+ 316us) server_negotiation.cc:407] Connection header received
0620 18:38:08.412765 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.412767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.412780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.412855 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.413844 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.414331 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.415495 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.415911 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.415985 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.416931 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.416932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.416933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:08.416937 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.416951 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:08.416955 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.417143 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.417280 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.417309 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:38:08.417624 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.411684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43469 (local address 127.10.109.65:52031)
0620 18:38:08.411917 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.411965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.412650 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.412942 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.412946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.412956 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:08.413691 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.413698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.414431 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.414432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.415399 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.415402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.416024 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.416025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.416186 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.416760 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:08.416761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:08.416800 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:08.416828 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:08.417150 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:08.417157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:08.417160 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.417281 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.417384 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260620 18:38:08.418124 28391 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43469
I20260620 18:38:08.418238 28391 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:08.418450 28391 heartbeater.cc:511] Master 127.10.109.126:43469 requested a full tablet report, sending...
I20260620 18:38:08.419008 28203 ts_manager.cc:194] Registered new tserver with Master: 4b3857b4fd504c05854a06c964f66349 (127.10.109.65:42697)
I20260620 18:38:08.419788 28203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.65:52031
W20260620 18:38:08.497835 28394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:08.498034 28394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:08.498067 28394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:08.499298 28394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:08.499341 28394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:08.499370 28394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:08.500726 28394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:08.500963 28394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:08.501297 28394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:08.506649 28402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.506705 28399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.506640 28400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.506966 28394 server_base.cc:1034] running on GCE node
I20260620 18:38:08.507197 28394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:08.507427 28394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:08.508606 28394 hybrid_clock.cc:648] HybridClock initialized: now 1781980688508607 us; error 43 us; skew 500 ppm
I20260620 18:38:08.509935 28394 webserver.cc:460] Webserver started at http://127.10.109.66:46553/ using document root <none> and password file <none>
I20260620 18:38:08.510200 28394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:08.510270 28394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:08.510418 28394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:08.511539 28394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "55eedafc653549bc8ef83a6316d240fa"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "743c1118ecdf794311f5b98bd9592e1f"
server_key_iv: "51baea5c9df7a99c1708e6215acddd67"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.511888 28394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "55eedafc653549bc8ef83a6316d240fa"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "743c1118ecdf794311f5b98bd9592e1f"
server_key_iv: "51baea5c9df7a99c1708e6215acddd67"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.513368 28394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:38:08.514447 28408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.514786 28394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:08.514966 28394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "55eedafc653549bc8ef83a6316d240fa"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "743c1118ecdf794311f5b98bd9592e1f"
server_key_iv: "51baea5c9df7a99c1708e6215acddd67"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.515069 28394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:08.536602 28394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:08.536957 28394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:08.537122 28394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:08.537361 28394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:08.537760 28394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:08.537814 28394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.537855 28394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:08.537882 28394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.543963 28394 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:35609
I20260620 18:38:08.544032 28521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:35609 every 8 connection(s)
I20260620 18:38:08.544337 28394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:08.549305 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28394
I20260620 18:38:08.549425 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:08.549726 10677 external_mini_cluster.cc:1428] Setting key 5e163b32c6f553693bdf93a1f3730435
I20260620 18:38:08.550493 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.545171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:55723 (local address 127.10.109.126:43469)
0620 18:38:08.545235 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.545238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.545403 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:38:08.545934 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.545935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.545947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.546004 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.546678 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.547209 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.548556 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.548798 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.548905 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.550187 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.550188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.550188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:08.550193 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.550213 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:08.550217 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.550273 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.550381 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.550415 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:38:08.550719 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.545046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43469 (local address 127.10.109.66:55723)
0620 18:38:08.545256 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.545300 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.545828 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.546069 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.546073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.546082 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:08.546582 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.546586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.547302 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.547303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.548359 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.548366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.548976 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.548977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.549144 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.549944 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:08.549944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:08.549974 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:08.550001 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:08.550286 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:08.550292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:08.550295 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.550384 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.550501 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260620 18:38:08.550823 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--builtin_ntp_servers=127.10.109.84:39173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:38:08.551446 28522 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43469
I20260620 18:38:08.551543 28522 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:08.551745 28522 heartbeater.cc:511] Master 127.10.109.126:43469 requested a full tablet report, sending...
I20260620 18:38:08.552105 28203 ts_manager.cc:194] Registered new tserver with Master: 55eedafc653549bc8ef83a6316d240fa (127.10.109.66:35609)
I20260620 18:38:08.552491 28203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.66:55723
W20260620 18:38:08.632951 28525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:08.633150 28525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:08.633172 28525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:08.634408 28525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:08.634447 28525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:08.634472 28525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:08.635950 28525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:39173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:08.636188 28525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:08.636471 28525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:08.641557 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.641532 28531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:08.642114 28530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:08.642220 28525 server_base.cc:1034] running on GCE node
I20260620 18:38:08.642477 28525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:08.642733 28525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:08.643954 28525 hybrid_clock.cc:648] HybridClock initialized: now 1781980688644002 us; error 96 us; skew 500 ppm
I20260620 18:38:08.645360 28525 webserver.cc:460] Webserver started at http://127.10.109.67:44033/ using document root <none> and password file <none>
I20260620 18:38:08.645640 28525 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:08.645745 28525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:08.645963 28525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:08.647069 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "696f164aa1db47a4901c4c7baea99aa8"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "bd7833619d226d1a5277be14642f9ed3"
server_key_iv: "7bf5526dc75856374ee4677830073099"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.647418 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "696f164aa1db47a4901c4c7baea99aa8"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "bd7833619d226d1a5277be14642f9ed3"
server_key_iv: "7bf5526dc75856374ee4677830073099"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.648680 28525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:08.649626 28539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.649942 28525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:08.650058 28525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "696f164aa1db47a4901c4c7baea99aa8"
format_stamp: "Formatted at 2026-06-20 18:38:08 on dist-test-slave-3jql"
server_key: "bd7833619d226d1a5277be14642f9ed3"
server_key_iv: "7bf5526dc75856374ee4677830073099"
server_key_version: "encryptionkey@0"
I20260620 18:38:08.650141 28525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:08.664170 28525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:08.664584 28525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:08.664734 28525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:08.665025 28525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:08.665383 28525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:08.665441 28525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.665484 28525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:08.665511 28525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:08.671278 28525 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:42193
I20260620 18:38:08.671326 28652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:42193 every 8 connection(s)
I20260620 18:38:08.671653 28525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:08.676517 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28525
I20260620 18:38:08.676626 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:08.676836 10677 external_mini_cluster.cc:1428] Setting key 9752194bb7084730785d943e4e05b4f9
I20260620 18:38:08.677270 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.672535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:44935 (local address 127.10.109.126:43469)
0620 18:38:08.672636 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.672638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.672832 (+ 194us) server_negotiation.cc:407] Connection header received
0620 18:38:08.673381 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.673383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.673396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.673452 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.674151 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.674682 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.675715 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.675906 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.675990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.676962 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.676964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.676964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:08.676969 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.676986 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:08.676991 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.677051 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.677167 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.677193 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:38:08.677489 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.672382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:43469 (local address 127.10.109.67:44935)
0620 18:38:08.672671 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.672713 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.673263 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.673540 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.673546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.673559 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:08.674049 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.674054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.674806 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.674807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.675593 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.675595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.676000 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.676001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.676138 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.676787 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:08.676788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:08.676814 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:08.676837 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:08.677071 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:08.677077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:08.677080 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.677155 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.677266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":115,"threads_started":1}
I20260620 18:38:08.677986 28653 heartbeater.cc:347] Connected to a master server at 127.10.109.126:43469
I20260620 18:38:08.678083 28653 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:08.678264 28653 heartbeater.cc:511] Master 127.10.109.126:43469 requested a full tablet report, sending...
I20260620 18:38:08.678632 28202 ts_manager.cc:194] Registered new tserver with Master: 696f164aa1db47a4901c4c7baea99aa8 (127.10.109.67:42193)
I20260620 18:38:08.679033 28202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.67:44935
I20260620 18:38:08.689350 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:08.694110 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.691026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.10.109.126:43469)
0620 18:38:08.691093 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.691095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.691166 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:38:08.691259 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.691260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.691279 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.691345 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.691475 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.691950 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.692908 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.693085 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.693147 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.693926 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.693927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.693927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:08.693931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.693943 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:08.693947 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.693993 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.694031 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.694050 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:38:08.696092 28202 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:08.697674 28202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:08.704217 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.700472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.10.109.65:42697)
0620 18:38:08.700662 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.700665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.700674 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:08.700709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.700711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.700726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.700809 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.700976 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.702050 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.703147 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.703780 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.703852 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.703936 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.704089 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.704125 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260620 18:38:08.704203 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.700206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42697 (local address 127.0.0.1:49718)
0620 18:38:08.700461 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.700472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.700528 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.700869 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.700870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.700872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.700904 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.700905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.702111 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.702112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.703054 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.703057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.703869 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.703871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.703951 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.703965 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.704066 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.704122 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260620 18:38:08.704586 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.699961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35609 (local address 127.0.0.1:39220)
0620 18:38:08.700140 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.700184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.700258 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.700664 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.700667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.700670 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.700732 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.700733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.702051 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.702052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.703339 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.703343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.704320 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.704321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.704386 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.704390 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.704475 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.704517 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":33,"thread_start_us":60,"threads_started":1}
I20260620 18:38:08.704675 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.700044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.10.109.66:35609)
0620 18:38:08.700233 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.700238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.700248 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:08.700340 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.700342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.700356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.700432 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.700827 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.701958 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.703430 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.704123 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.704189 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.704280 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.704520 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.704568 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260620 18:38:08.704953 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 44c8a3e35e4c4112bda6868874255813 (DEFAULT_TABLE table=test-table [id=e183baa57d1143aa8ac963fd5890ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.705331 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c8a3e35e4c4112bda6868874255813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.705284 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 44c8a3e35e4c4112bda6868874255813 (DEFAULT_TABLE table=test-table [id=e183baa57d1143aa8ac963fd5890ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.705526 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.700133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.10.109.67:42193)
0620 18:38:08.700539 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.700544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.700555 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:08.700596 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.700598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.700614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.700693 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.700920 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.701677 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.703558 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.704253 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.704991 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.705093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.705390 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.705433 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":100,"threads_started":1}
I20260620 18:38:08.705585 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c8a3e35e4c4112bda6868874255813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.705785 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.699996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42193 (local address 127.0.0.1:33410)
0620 18:38:08.700325 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.700336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.700396 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.700782 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.700784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.700786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.700835 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.700836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.701791 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.701792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.703467 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.703469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.705197 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.705198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.705273 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.705276 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.705484 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.705535 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":74,"thread_start_us":93,"threads_started":1}
I20260620 18:38:08.706430 28587 tablet_service.cc:1467] Processing CreateTablet for tablet 44c8a3e35e4c4112bda6868874255813 (DEFAULT_TABLE table=test-table [id=e183baa57d1143aa8ac963fd5890ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:08.706799 28587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c8a3e35e4c4112bda6868874255813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:08.708938 28667 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Bootstrap starting.
I20260620 18:38:08.709619 28667 tablet_bootstrap.cc:654] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.709931 28667 log.cc:824] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.710278 28668 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: Bootstrap starting.
I20260620 18:38:08.710879 28669 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Bootstrap starting.
I20260620 18:38:08.710940 28668 tablet_bootstrap.cc:654] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.711138 28667 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: No bootstrap required, opened a new log
I20260620 18:38:08.711265 28667 ts_tablet_manager.cc:1397] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:08.711239 28668 log.cc:824] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.711661 28669 tablet_bootstrap.cc:654] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:08.711968 28669 log.cc:824] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:08.712097 28668 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: No bootstrap required, opened a new log
I20260620 18:38:08.712190 28668 ts_tablet_manager.cc:1397] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:08.712661 28669 tablet_bootstrap.cc:492] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: No bootstrap required, opened a new log
I20260620 18:38:08.712738 28669 ts_tablet_manager.cc:1397] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:08.714196 28667 raft_consensus.cc:348] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.714395 28667 raft_consensus.cc:374] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.714442 28667 raft_consensus.cc:729] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55eedafc653549bc8ef83a6316d240fa, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.714569 28667 consensus_queue.cc:260] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.714864 28667 ts_tablet_manager.cc:1428] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260620 18:38:08.714811 28669 raft_consensus.cc:348] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.714937 28669 raft_consensus.cc:374] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.714982 28669 raft_consensus.cc:729] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696f164aa1db47a4901c4c7baea99aa8, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.714993 28522 heartbeater.cc:503] Master 127.10.109.126:43469 was elected leader, sending a full tablet report...
I20260620 18:38:08.715006 28668 raft_consensus.cc:348] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.715097 28669 consensus_queue.cc:260] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.715167 28668 raft_consensus.cc:374] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:08.715340 28669 ts_tablet_manager.cc:1428] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:08.715468 28653 heartbeater.cc:503] Master 127.10.109.126:43469 was elected leader, sending a full tablet report...
I20260620 18:38:08.715448 28668 raft_consensus.cc:729] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3857b4fd504c05854a06c964f66349, State: Initialized, Role: FOLLOWER
I20260620 18:38:08.715590 28668 consensus_queue.cc:260] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.715868 28391 heartbeater.cc:503] Master 127.10.109.126:43469 was elected leader, sending a full tablet report...
I20260620 18:38:08.715903 28668 ts_tablet_manager.cc:1428] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:38:08.753069 28675 raft_consensus.cc:482] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:08.753410 28675 raft_consensus.cc:504] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.753914 28675 leader_election.cc:288] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55eedafc653549bc8ef83a6316d240fa (127.10.109.66:35609), 696f164aa1db47a4901c4c7baea99aa8 (127.10.109.67:42193)
I20260620 18:38:08.757730 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.754051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:35609 (local address 127.10.109.65:32929)
0620 18:38:08.754105 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.754117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.754159 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.754332 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.754333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.754335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.754414 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.754416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.755453 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.755454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.756655 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.756658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.757435 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.757436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.757523 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.757527 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.757590 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.757635 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260620 18:38:08.757736 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.754126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42193 (local address 127.10.109.65:39887)
0620 18:38:08.754360 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.754370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:08.754434 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:08.754618 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:08.754619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:08.754620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:08.754683 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.754685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.755582 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.755583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.756674 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:08.756676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.757435 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.757436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:08.757523 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.757528 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:38:08.757590 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:38:08.757634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":8,"thread_start_us":152,"threads_started":1}
I20260620 18:38:08.757866 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.754125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:32929 (local address 127.10.109.66:35609)
0620 18:38:08.754165 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.754167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.754193 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:38:08.754220 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.754221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.754231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.754295 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.754518 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.755352 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.756774 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.757330 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.757412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.757420 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.757690 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.757772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:38:08.758064 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.754210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:39887 (local address 127.10.109.67:42193)
0620 18:38:08.754296 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.754298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.754492 (+ 194us) server_negotiation.cc:407] Connection header received
0620 18:38:08.754527 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.754528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.754540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.754605 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:08.754776 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.755498 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.756774 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.757333 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.757412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.757420 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.757690 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.757772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:08.758253 28476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c8a3e35e4c4112bda6868874255813" candidate_uuid: "4b3857b4fd504c05854a06c964f66349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55eedafc653549bc8ef83a6316d240fa" is_pre_election: true
I20260620 18:38:08.758409 28476 raft_consensus.cc:2454] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3857b4fd504c05854a06c964f66349 in term 0.
I20260620 18:38:08.758423 28607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c8a3e35e4c4112bda6868874255813" candidate_uuid: "4b3857b4fd504c05854a06c964f66349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f164aa1db47a4901c4c7baea99aa8" is_pre_election: true
I20260620 18:38:08.758673 28607 raft_consensus.cc:2454] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3857b4fd504c05854a06c964f66349 in term 0.
I20260620 18:38:08.758720 28280 leader_election.cc:302] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3857b4fd504c05854a06c964f66349, 55eedafc653549bc8ef83a6316d240fa; no voters:
I20260620 18:38:08.758885 28675 raft_consensus.cc:2781] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:08.758955 28675 raft_consensus.cc:482] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:08.758986 28675 raft_consensus.cc:3037] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.759797 28675 raft_consensus.cc:504] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.759953 28675 leader_election.cc:288] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [CANDIDATE]: Term 1 election: Requested vote from peers 55eedafc653549bc8ef83a6316d240fa (127.10.109.66:35609), 696f164aa1db47a4901c4c7baea99aa8 (127.10.109.67:42193)
I20260620 18:38:08.760128 28607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c8a3e35e4c4112bda6868874255813" candidate_uuid: "4b3857b4fd504c05854a06c964f66349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f164aa1db47a4901c4c7baea99aa8"
I20260620 18:38:08.760257 28607 raft_consensus.cc:3037] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.760296 28476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c8a3e35e4c4112bda6868874255813" candidate_uuid: "4b3857b4fd504c05854a06c964f66349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55eedafc653549bc8ef83a6316d240fa"
I20260620 18:38:08.760401 28476 raft_consensus.cc:3037] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:08.761124 28607 raft_consensus.cc:2454] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3857b4fd504c05854a06c964f66349 in term 1.
I20260620 18:38:08.761319 28281 leader_election.cc:302] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3857b4fd504c05854a06c964f66349, 696f164aa1db47a4901c4c7baea99aa8; no voters:
I20260620 18:38:08.761400 28476 raft_consensus.cc:2454] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3857b4fd504c05854a06c964f66349 in term 1.
I20260620 18:38:08.761466 28675 raft_consensus.cc:2781] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:08.761615 28675 raft_consensus.cc:686] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3857b4fd504c05854a06c964f66349, State: Running, Role: LEADER
I20260620 18:38:08.761723 28675 consensus_queue.cc:237] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } }
I20260620 18:38:08.762611 28201 catalog_manager.cc:5495] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3857b4fd504c05854a06c964f66349 (127.10.109.65). New cstate: current_term: 1 leader_uuid: "4b3857b4fd504c05854a06c964f66349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3857b4fd504c05854a06c964f66349" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 42697 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:08.780526 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.777879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.10.109.65:42697)
0620 18:38:08.777925 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.777927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.777937 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:08.778010 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.778011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.778038 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.778121 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:08.778276 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.778853 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.779657 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.779874 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.779968 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.780159 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:08.780265 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:08.780366 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.780388 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.780410 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 18:38:08.782407 28607 raft_consensus.cc:1261] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 1 FOLLOWER]: Refusing update from remote peer 4b3857b4fd504c05854a06c964f66349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:08.782712 28476 raft_consensus.cc:1261] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 1 FOLLOWER]: Refusing update from remote peer 4b3857b4fd504c05854a06c964f66349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:08.782841 28677 consensus_queue.cc:1035] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696f164aa1db47a4901c4c7baea99aa8" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:08.783049 28677 consensus_queue.cc:1035] T 44c8a3e35e4c4112bda6868874255813 P 4b3857b4fd504c05854a06c964f66349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55eedafc653549bc8ef83a6316d240fa" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:08.785550 28681 mvcc.cc:205] Tried to move back new op lower bound from 7298992901249658880 to 7298992901168586752. Current Snapshot: MvccSnapshot[applied={T|T < 7298992901249658880}]
I20260620 18:38:08.786058 28682 mvcc.cc:205] Tried to move back new op lower bound from 7298992901249658880 to 7298992901168586752. Current Snapshot: MvccSnapshot[applied={T|T < 7298992901249658880}]
I20260620 18:38:08.788273 28201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:08.788364 28201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:08.789089 28201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf6e9105c09427bb1e47a38314bfd70: Sending DeleteTablet for 3 replicas of tablet 44c8a3e35e4c4112bda6868874255813
I20260620 18:38:08.789425 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c8a3e35e4c4112bda6868874255813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='slave'} at 127.0.0.1:39220
I20260620 18:38:08.789495 28587 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c8a3e35e4c4112bda6868874255813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='slave'} at 127.0.0.1:33410
I20260620 18:38:08.789646 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28263
I20260620 18:38:08.789686 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c8a3e35e4c4112bda6868874255813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:08 UTC) from {username='slave'} at 127.0.0.1:49718
I20260620 18:38:08.790387 28689 tablet_replica.cc:330] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: stopping tablet replica
I20260620 18:38:08.790503 28689 raft_consensus.cc:2229] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:08.790625 28689 raft_consensus.cc:2258] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:08.790652 28690 tablet_replica.cc:330] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: stopping tablet replica
I20260620 18:38:08.790975 28690 raft_consensus.cc:2229] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:08.791082 28690 raft_consensus.cc:2258] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:08.791324 28689 ts_tablet_manager.cc:1905] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:08.791463 28690 ts_tablet_manager.cc:1905] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:08.793211 28689 ts_tablet_manager.cc:1918] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:08.793315 28690 ts_tablet_manager.cc:1918] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:08.793382 28690 log.cc:1196] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/44c8a3e35e4c4112bda6868874255813
I20260620 18:38:08.793402 28689 log.cc:1196] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/44c8a3e35e4c4112bda6868874255813
I20260620 18:38:08.793664 28690 ts_tablet_manager.cc:1939] T 44c8a3e35e4c4112bda6868874255813 P 696f164aa1db47a4901c4c7baea99aa8: Deleting consensus metadata
I20260620 18:38:08.793776 28689 ts_tablet_manager.cc:1939] T 44c8a3e35e4c4112bda6868874255813 P 55eedafc653549bc8ef83a6316d240fa: Deleting consensus metadata
I20260620 18:38:08.794068 28190 catalog_manager.cc:4841] TS 696f164aa1db47a4901c4c7baea99aa8 (127.10.109.67:42193): tablet 44c8a3e35e4c4112bda6868874255813 (table test-table [id=e183baa57d1143aa8ac963fd5890ed02]) successfully deleted
I20260620 18:38:08.794303 28189 catalog_manager.cc:4841] TS 55eedafc653549bc8ef83a6316d240fa (127.10.109.66:35609): tablet 44c8a3e35e4c4112bda6868874255813 (table test-table [id=e183baa57d1143aa8ac963fd5890ed02]) successfully deleted
W20260620 18:38:08.795948 28190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:42697 (error 108)
I20260620 18:38:08.796482 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.796332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:42697 (local address 127.0.0.1:49734)
0620 18:38:08.796402 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:08.796452 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:38:08.796525 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28394
W20260620 18:38:08.796640 28190 catalog_manager.cc:4584] TS 4b3857b4fd504c05854a06c964f66349 (127.10.109.65:42697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44c8a3e35e4c4112bda6868874255813: Network error: Client connection negotiation failed: client connection to 127.10.109.65:42697: connect: Connection refused (error 111)
I20260620 18:38:08.801373 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28525
I20260620 18:38:08.806229 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28172
2026-06-20T18:38:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (698 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:08 dist-test-slave-3jql krb5kdc[28694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:08 dist-test-slave-3jql krb5kdc[28694](info): set up 2 sockets
Jun 20 18:38:08 dist-test-slave-3jql krb5kdc[28694](info): commencing operation
krb5kdc: starting...
W20260620 18:38:09.866688 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:09 dist-test-slave-3jql krb5kdc[28694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:09.985659 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45555
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.070672 28707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.070951 28707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.070990 28707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.072120 28707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:10.072187 28707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.072223 28707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.072252 28707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:10.072276 28707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:10.073690 28707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.073954 28707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.074170 28707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.078840 28715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.078846 28712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.078847 28713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.078967 28707 server_base.cc:1034] running on GCE node
I20260620 18:38:10.079305 28707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.079588 28707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.080739 28707 hybrid_clock.cc:648] HybridClock initialized: now 1781980690080715 us; error 37 us; skew 500 ppm
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.083201 28707 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:10.083598 28707 webserver.cc:460] Webserver started at http://127.10.109.126:38531/ using document root <none> and password file <none>
I20260620 18:38:10.083809 28707 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.083853 28707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.084014 28707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.085080 28707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "fa5045cce3e443baba6be5ecfb1b0b46"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fd3459fb56481bff762814f8be4ed8ef"
server_key_iv: "4cc39e6970a9ff130e2c5432bd8115e3"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.085449 28707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "fa5045cce3e443baba6be5ecfb1b0b46"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fd3459fb56481bff762814f8be4ed8ef"
server_key_iv: "4cc39e6970a9ff130e2c5432bd8115e3"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.086720 28707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:10.087550 28722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.087733 28707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:10.087813 28707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "fa5045cce3e443baba6be5ecfb1b0b46"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fd3459fb56481bff762814f8be4ed8ef"
server_key_iv: "4cc39e6970a9ff130e2c5432bd8115e3"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.087865 28707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.104677 28707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.106894 28707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.107000 28707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.110627 28707 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:45555
I20260620 18:38:10.110697 28774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:45555 every 8 connection(s)
I20260620 18:38:10.111023 28707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:10.111896 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:10.114018 28775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: Bootstrap starting.
I20260620 18:38:10.114523 28775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:10.114832 28775 log.cc:824] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:10.115507 28775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: No bootstrap required, opened a new log
I20260620 18:38:10.117053 28775 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } }
I20260620 18:38:10.117189 28775 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:10.117218 28775 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5045cce3e443baba6be5ecfb1b0b46, State: Initialized, Role: FOLLOWER
I20260620 18:38:10.117369 28775 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } }
I20260620 18:38:10.117455 28775 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:10.117477 28775 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:10.117506 28775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:10.117997 28775 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } }
I20260620 18:38:10.118081 28775 leader_election.cc:302] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5045cce3e443baba6be5ecfb1b0b46; no voters:
I20260620 18:38:10.118201 28775 leader_election.cc:288] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:10.118287 28778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:10.118431 28778 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [term 1 LEADER]: Becoming Leader. State: Replica: fa5045cce3e443baba6be5ecfb1b0b46, State: Running, Role: LEADER
I20260620 18:38:10.118450 28775 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:10.118515 28778 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } }
I20260620 18:38:10.118952 28780 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5045cce3e443baba6be5ecfb1b0b46. Latest consensus state: current_term: 1 leader_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } } }
I20260620 18:38:10.119051 28780 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:10.119068 28779 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5045cce3e443baba6be5ecfb1b0b46" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45555 } } }
I20260620 18:38:10.119118 28779 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:10.119376 28787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:10.119881 28787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:10.120154 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28707
I20260620 18:38:10.120265 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:10.120589 10677 external_mini_cluster.cc:1428] Setting key d71e73d17c6231d55c023ed29464f2c5
I20260620 18:38:10.121475 28787 catalog_manager.cc:1347] Generated new cluster ID: 7eea06b820ed4b2c85a7f48c908269e1
I20260620 18:38:10.121526 28787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:10.131930 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.121050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.10.109.126:45555)
0620 18:38:10.121245 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.121254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.121477 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:38:10.121585 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.121592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.121653 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.121782 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.122487 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.123923 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.124939 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.125574 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.125678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.128449 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.128453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.128453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.128464 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.130317 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.130700 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.130702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.130702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.130826 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.131035 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.131036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.131036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.131311 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.131459 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.131594 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.131715 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260620 18:38:10.137815 28787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:10.138326 28787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:10.147672 28787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: Generated new TSK 0
I20260620 18:38:10.147854 28787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:38:10.181735 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.261548 28801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.261780 28801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.261814 28801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.262933 28801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.262998 28801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.263043 28801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:10.264521 28801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.264799 28801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.265023 28801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.270115 28809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.270057 28807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.270074 28806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.270355 28801 server_base.cc:1034] running on GCE node
I20260620 18:38:10.270581 28801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.270859 28801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.272011 28801 hybrid_clock.cc:648] HybridClock initialized: now 1781980690271979 us; error 50 us; skew 500 ppm
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.274479 28801 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:38:10.274981 28801 webserver.cc:460] Webserver started at http://127.10.109.65:46403/ using document root <none> and password file <none>
I20260620 18:38:10.275218 28801 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.275305 28801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.275435 28801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.276464 28801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "4e2f8a7a58574be180eb568e3c630910"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "31f6a92c2347d51a5755bb251f5399fd"
server_key_iv: "e7b1bd1a9a50de8b876955bad767002f"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.276778 28801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "4e2f8a7a58574be180eb568e3c630910"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "31f6a92c2347d51a5755bb251f5399fd"
server_key_iv: "e7b1bd1a9a50de8b876955bad767002f"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.278014 28801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:10.278699 28816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.278878 28801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:10.278976 28801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "4e2f8a7a58574be180eb568e3c630910"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "31f6a92c2347d51a5755bb251f5399fd"
server_key_iv: "e7b1bd1a9a50de8b876955bad767002f"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.279059 28801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.292917 28801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.294720 28801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.294965 28801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.295187 28801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:10.295506 28801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:10.295562 28801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.295603 28801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:10.295629 28801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.301478 28801 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:45847
I20260620 18:38:10.301534 28929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:45847 every 8 connection(s)
I20260620 18:38:10.301828 28801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:10.306342 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28801
I20260620 18:38:10.306473 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:10.306684 10677 external_mini_cluster.cc:1428] Setting key 1bdc8306096dff307d7f910f3579b3d7
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:10.311956 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.302656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:54351 (local address 127.10.109.126:45555)
0620 18:38:10.302732 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.302734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.302940 (+ 206us) server_negotiation.cc:407] Connection header received
0620 18:38:10.303504 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.303506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.303523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.303584 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.304578 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.305103 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.306077 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.306312 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.306399 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.308929 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.308930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.308931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.308942 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.310462 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.310968 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.310969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.310970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.311029 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.311303 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.311304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.311304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.311424 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.311500 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.311776 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.311868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:38:10.312130 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.302529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45555 (local address 127.10.109.65:54351)
0620 18:38:10.302797 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.302847 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.303397 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.303652 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.303656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.304298 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.304477 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.304481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.305211 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.305212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.305980 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.305982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.306450 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.306451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.306595 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.307207 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.307217 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.308836 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.310556 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.310563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.310566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.310876 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.311104 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.311105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.311105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.311226 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.311492 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.311499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.311639 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.311742 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.311895 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260620 18:38:10.312668 28930 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45555
I20260620 18:38:10.312788 28930 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:10.312978 28930 heartbeater.cc:511] Master 127.10.109.126:45555 requested a full tablet report, sending...
I20260620 18:38:10.313478 28739 ts_manager.cc:194] Registered new tserver with Master: 4e2f8a7a58574be180eb568e3c630910 (127.10.109.65:45847)
I20260620 18:38:10.314075 28739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:54351
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:38:10.334332 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.414856 28935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.415048 28935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.415071 28935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.416145 28935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.416183 28935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.416216 28935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:10.417755 28935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.418025 28935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.418280 28935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.423225 28941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.423235 28940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.423234 28943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.423431 28935 server_base.cc:1034] running on GCE node
I20260620 18:38:10.423691 28935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.423923 28935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.425099 28935 hybrid_clock.cc:648] HybridClock initialized: now 1781980690425081 us; error 28 us; skew 500 ppm
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.427976 28935 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:38:10.428701 28935 webserver.cc:460] Webserver started at http://127.10.109.66:35305/ using document root <none> and password file <none>
I20260620 18:38:10.428994 28935 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.429050 28935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.429173 28935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.430215 28935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "c572af9fedc84ab8ad82f4d87b2f6109"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "366cac40dd6bbd5613877ef1490338d3"
server_key_iv: "ededdf898b153b83971cb7a65957c913"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.430567 28935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "c572af9fedc84ab8ad82f4d87b2f6109"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "366cac40dd6bbd5613877ef1490338d3"
server_key_iv: "ededdf898b153b83971cb7a65957c913"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.431951 28935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:10.432816 28950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.433041 28935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:10.433164 28935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "c572af9fedc84ab8ad82f4d87b2f6109"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "366cac40dd6bbd5613877ef1490338d3"
server_key_iv: "ededdf898b153b83971cb7a65957c913"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.433314 28935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.446417 28935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.448673 28935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.448841 28935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.449076 28935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:10.449378 28935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:10.449429 28935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.449471 28935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:10.449498 28935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.455219 28935 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:41399
I20260620 18:38:10.455267 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:41399 every 8 connection(s)
I20260620 18:38:10.455578 28935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:10.459285 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 28935
I20260620 18:38:10.459379 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:10.459604 10677 external_mini_cluster.cc:1428] Setting key 1c46866af741977c39ad54db632912f9
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:10.464943 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.456299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:55385 (local address 127.10.109.126:45555)
0620 18:38:10.456386 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.456389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.456549 (+ 160us) server_negotiation.cc:407] Connection header received
0620 18:38:10.457016 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.457017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.457027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.457091 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.457972 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.458509 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.459737 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.460006 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.460091 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.462168 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.462170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.462170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.462178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.463564 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.463985 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.463986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.463986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.464046 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.464302 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.464303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.464303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.464416 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.464493 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.464758 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.464855 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:38:10.465126 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.456246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45555 (local address 127.10.109.66:55385)
0620 18:38:10.456431 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.456465 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.456929 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.457132 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.457135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.457670 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.457864 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.457869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.458605 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.458606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.459612 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.459615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.460099 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.460100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.460244 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.460769 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.460777 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.462074 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.463661 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.463668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.463670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.463894 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.464115 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.464116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.464116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.464227 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.464485 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.464491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.464626 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.464733 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.464897 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260620 18:38:10.465552 29064 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45555
I20260620 18:38:10.465642 29064 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:10.465834 29064 heartbeater.cc:511] Master 127.10.109.126:45555 requested a full tablet report, sending...
I20260620 18:38:10.466204 28739 ts_manager.cc:194] Registered new tserver with Master: c572af9fedc84ab8ad82f4d87b2f6109 (127.10.109.66:41399)
I20260620 18:38:10.466707 28739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:55385
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:38:10.484944 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.565081 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.565294 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.565318 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.566569 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.566615 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.566644 29069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:10.568248 29069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.568480 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.568706 29069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.573599 29075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.573599 29077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.573786 29069 server_base.cc:1034] running on GCE node
W20260620 18:38:10.573609 29074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.574095 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.574329 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.575484 29069 hybrid_clock.cc:648] HybridClock initialized: now 1781980690575481 us; error 36 us; skew 500 ppm
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.577891 29069 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:38:10.578326 29069 webserver.cc:460] Webserver started at http://127.10.109.67:36561/ using document root <none> and password file <none>
I20260620 18:38:10.578559 29069 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.578627 29069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.578809 29069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.579793 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "ec862ed1661d4fd78bf679c9adc5fd1f"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "b2674257dd4e7e209f9175d0049e5696"
server_key_iv: "24704d62126ea911996a517d41541e4e"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.580137 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "ec862ed1661d4fd78bf679c9adc5fd1f"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "b2674257dd4e7e209f9175d0049e5696"
server_key_iv: "24704d62126ea911996a517d41541e4e"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.581463 29069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:10.582320 29084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.582548 29069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:10.582651 29069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "ec862ed1661d4fd78bf679c9adc5fd1f"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "b2674257dd4e7e209f9175d0049e5696"
server_key_iv: "24704d62126ea911996a517d41541e4e"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.582739 29069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.598328 29069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.600082 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.600239 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.600442 29069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:10.600730 29069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:10.600785 29069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.600816 29069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:10.600844 29069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.606288 29069 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:45511
I20260620 18:38:10.606360 29197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:45511 every 8 connection(s)
I20260620 18:38:10.606794 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:10.609367 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29069
I20260620 18:38:10.609532 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:10.609730 10677 external_mini_cluster.cc:1428] Setting key 984d687df764540ab5bb5ffa2eb47cbc
Jun 20 18:38:10 dist-test-slave-3jql krb5kdc[28694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:10.615723 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.607461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:39983 (local address 127.10.109.126:45555)
0620 18:38:10.607500 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.607502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.607708 (+ 206us) server_negotiation.cc:407] Connection header received
0620 18:38:10.608236 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.608238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.608250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.608312 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.609176 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.609791 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.611098 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.611314 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.611397 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.613283 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.613284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.613284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.613290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.614434 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.614840 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.614841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.614841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.614891 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.615107 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.615108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.615108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.615206 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.615298 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.615604 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.615664 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:10.615916 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.607428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45555 (local address 127.10.109.67:39983)
0620 18:38:10.607596 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.607633 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.608132 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.608374 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.608378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.608908 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.609076 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.609080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.609881 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.609882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.611012 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.611015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.611407 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.611408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.611538 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.611918 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.611925 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.613194 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.614503 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.614509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.614510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.614732 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.614949 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.614949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.614949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.615039 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.615320 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.615328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.615475 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.615575 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.615685 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260620 18:38:10.616372 29198 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45555
I20260620 18:38:10.616446 29198 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:10.616621 29198 heartbeater.cc:511] Master 127.10.109.126:45555 requested a full tablet report, sending...
I20260620 18:38:10.616981 28739 ts_manager.cc:194] Registered new tserver with Master: ec862ed1661d4fd78bf679c9adc5fd1f (127.10.109.67:45511)
I20260620 18:38:10.617355 28739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:39983
I20260620 18:38:10.621268 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:10.628515 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.622501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.10.109.126:45555)
0620 18:38:10.622555 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.622557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.622665 (+ 108us) server_negotiation.cc:407] Connection header received
0620 18:38:10.622751 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.622752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.622763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.622831 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.623369 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.623809 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.624540 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.624718 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.624805 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.626046 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.626048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.626048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.626055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.627338 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.627714 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.627715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.627715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.627770 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.627992 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.627993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.627994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.628118 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.628230 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.628355 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.628416 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:10.630081 28739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:10.631651 28739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:10.637410 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.633690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:41399 (local address 127.0.0.1:51402)
0620 18:38:10.633831 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.633866 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.633927 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.634198 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.634201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.634204 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:10.634291 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.634293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635400 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.635401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.636376 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.636379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637126 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.637209 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.637224 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.637291 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.637340 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":49,"threads_started":1}
I20260620 18:38:10.638162 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.633845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.10.109.66:41399)
0620 18:38:10.634017 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.634021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.634031 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:10.634072 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.634074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.634090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.634171 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:10.634376 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635257 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.636466 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637043 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637113 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.637168 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.638038 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.638081 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260620 18:38:10.638188 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.633890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.10.109.67:45511)
0620 18:38:10.634113 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.634117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.634476 (+ 359us) server_negotiation.cc:407] Connection header received
0620 18:38:10.634512 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.634515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.634528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.634595 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:10.634794 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635837 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637270 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637845 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637904 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.637973 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.638070 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.638112 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":147,"threads_started":1}
I20260620 18:38:10.638254 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.633690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:45847 (local address 127.0.0.1:58772)
0620 18:38:10.634213 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.634224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.634276 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.634532 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.634533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.634535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:10.634590 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.634591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635950 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.635952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.637165 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.637167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637960 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.638036 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.638042 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.638095 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.638136 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":42,"thread_start_us":49,"threads_started":1}
I20260620 18:38:10.638162 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.633730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:45511 (local address 127.0.0.1:36052)
0620 18:38:10.634368 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.634377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.634420 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.634652 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.634652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.634654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:10.634695 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.634696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635957 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.635958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.637164 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.637167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637933 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.637992 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.637996 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.638057 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.638096 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":41,"thread_start_us":53,"threads_started":1}
I20260620 18:38:10.638906 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.634029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.10.109.65:45847)
0620 18:38:10.634202 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.634206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.634266 (+ 60us) server_negotiation.cc:407] Connection header received
0620 18:38:10.634343 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.634345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.634381 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.634454 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:10.634669 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.635824 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637237 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.637852 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.637944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.637995 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.638202 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.638848 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260620 18:38:10.638825 28998 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf236b4b284d39ad237b2f126bcecb (DEFAULT_TABLE table=test-table [id=fe3dbf98b375427188487b18412da4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:10.639178 28998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf236b4b284d39ad237b2f126bcecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:10.639259 29132 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf236b4b284d39ad237b2f126bcecb (DEFAULT_TABLE table=test-table [id=fe3dbf98b375427188487b18412da4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:10.639539 29132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf236b4b284d39ad237b2f126bcecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:10.639539 28864 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf236b4b284d39ad237b2f126bcecb (DEFAULT_TABLE table=test-table [id=fe3dbf98b375427188487b18412da4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:10.639824 28864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf236b4b284d39ad237b2f126bcecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:10.642306 29212 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: Bootstrap starting.
I20260620 18:38:10.642792 29213 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Bootstrap starting.
I20260620 18:38:10.643092 29212 tablet_bootstrap.cc:654] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:10.643445 29213 tablet_bootstrap.cc:654] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:10.643674 29212 log.cc:824] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:10.643982 29213 log.cc:824] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:10.644292 29214 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Bootstrap starting.
I20260620 18:38:10.644477 29212 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: No bootstrap required, opened a new log
I20260620 18:38:10.644554 29212 ts_tablet_manager.cc:1397] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:10.644856 29214 tablet_bootstrap.cc:654] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:10.645390 29214 log.cc:824] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:10.645892 29213 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: No bootstrap required, opened a new log
I20260620 18:38:10.645973 29213 ts_tablet_manager.cc:1397] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:10.646117 29214 tablet_bootstrap.cc:492] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: No bootstrap required, opened a new log
I20260620 18:38:10.646191 29214 ts_tablet_manager.cc:1397] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:10.646137 29212 raft_consensus.cc:348] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.646327 29212 raft_consensus.cc:374] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:10.646368 29212 raft_consensus.cc:729] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2f8a7a58574be180eb568e3c630910, State: Initialized, Role: FOLLOWER
I20260620 18:38:10.646476 29212 consensus_queue.cc:260] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.646772 29212 ts_tablet_manager.cc:1428] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:38:10.646894 28930 heartbeater.cc:503] Master 127.10.109.126:45555 was elected leader, sending a full tablet report...
I20260620 18:38:10.648048 29213 raft_consensus.cc:348] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.648208 29213 raft_consensus.cc:374] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:10.648259 29213 raft_consensus.cc:729] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c572af9fedc84ab8ad82f4d87b2f6109, State: Initialized, Role: FOLLOWER
I20260620 18:38:10.648177 29214 raft_consensus.cc:348] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.648358 29214 raft_consensus.cc:374] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:10.648368 29213 consensus_queue.cc:260] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.648406 29214 raft_consensus.cc:729] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec862ed1661d4fd78bf679c9adc5fd1f, State: Initialized, Role: FOLLOWER
I20260620 18:38:10.648512 29214 consensus_queue.cc:260] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.648613 29213 ts_tablet_manager.cc:1428] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:10.648643 29064 heartbeater.cc:503] Master 127.10.109.126:45555 was elected leader, sending a full tablet report...
I20260620 18:38:10.648798 29214 ts_tablet_manager.cc:1428] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:38:10.648844 29198 heartbeater.cc:503] Master 127.10.109.126:45555 was elected leader, sending a full tablet report...
I20260620 18:38:10.671640 29219 raft_consensus.cc:482] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:10.671722 29219 raft_consensus.cc:504] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.672081 29219 leader_election.cc:288] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e2f8a7a58574be180eb568e3c630910 (127.10.109.65:45847), ec862ed1661d4fd78bf679c9adc5fd1f (127.10.109.67:45511)
I20260620 18:38:10.674871 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.672277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:60209 (local address 127.10.109.65:45847)
0620 18:38:10.672318 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.672320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.672329 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:10.672400 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.672401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.672412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.672468 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:10.672609 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.673163 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674131 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.674513 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674567 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.674573 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.674768 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.674810 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:38:10.674872 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.672200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:45847 (local address 127.10.109.66:60209)
0620 18:38:10.672277 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.672288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.672331 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.672470 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.672470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.672472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:10.672536 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.672538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.673281 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.673282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.674033 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.674035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.674590 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.674675 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.674678 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.674720 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.674799 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":36}
I20260620 18:38:10.674921 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.672320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:45535 (local address 127.10.109.67:45511)
0620 18:38:10.672375 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.672376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.672440 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:38:10.672488 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.672488 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.672497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.672563 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:10.672753 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.673166 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674181 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.674524 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674613 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.674621 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.674817 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.674855 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:10.674953 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.672217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:45511 (local address 127.10.109.66:45535)
0620 18:38:10.672367 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.672378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.672439 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.672616 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.672616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.672618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:10.672681 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.672683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.673252 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.673253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.674065 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.674090 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.674640 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.674641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.674696 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.674699 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.674796 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.674834 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260620 18:38:10.675227 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf236b4b284d39ad237b2f126bcecb" candidate_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" is_pre_election: true
I20260620 18:38:10.675225 28884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf236b4b284d39ad237b2f126bcecb" candidate_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2f8a7a58574be180eb568e3c630910" is_pre_election: true
I20260620 18:38:10.675382 29152 raft_consensus.cc:2454] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c572af9fedc84ab8ad82f4d87b2f6109 in term 0.
I20260620 18:38:10.675396 28884 raft_consensus.cc:2454] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c572af9fedc84ab8ad82f4d87b2f6109 in term 0.
I20260620 18:38:10.675608 28953 leader_election.cc:302] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2f8a7a58574be180eb568e3c630910, c572af9fedc84ab8ad82f4d87b2f6109; no voters:
I20260620 18:38:10.675755 29219 raft_consensus.cc:2781] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:10.675802 29219 raft_consensus.cc:482] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:10.675825 29219 raft_consensus.cc:3037] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:10.676436 29219 raft_consensus.cc:504] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.676576 29219 leader_election.cc:288] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [CANDIDATE]: Term 1 election: Requested vote from peers 4e2f8a7a58574be180eb568e3c630910 (127.10.109.65:45847), ec862ed1661d4fd78bf679c9adc5fd1f (127.10.109.67:45511)
I20260620 18:38:10.676712 28884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf236b4b284d39ad237b2f126bcecb" candidate_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2f8a7a58574be180eb568e3c630910"
I20260620 18:38:10.676792 28884 raft_consensus.cc:3037] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:10.677469 28884 raft_consensus.cc:2454] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c572af9fedc84ab8ad82f4d87b2f6109 in term 1.
I20260620 18:38:10.677598 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf236b4b284d39ad237b2f126bcecb" candidate_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f"
I20260620 18:38:10.677641 28953 leader_election.cc:302] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2f8a7a58574be180eb568e3c630910, c572af9fedc84ab8ad82f4d87b2f6109; no voters:
I20260620 18:38:10.677670 29152 raft_consensus.cc:3037] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:10.677727 29219 raft_consensus.cc:2781] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:10.677858 29219 raft_consensus.cc:686] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 1 LEADER]: Becoming Leader. State: Replica: c572af9fedc84ab8ad82f4d87b2f6109, State: Running, Role: LEADER
I20260620 18:38:10.677940 29219 consensus_queue.cc:237] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } }
I20260620 18:38:10.678485 29152 raft_consensus.cc:2454] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c572af9fedc84ab8ad82f4d87b2f6109 in term 1.
I20260620 18:38:10.678681 28739 catalog_manager.cc:5495] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 reported cstate change: term changed from 0 to 1, leader changed from <none> to c572af9fedc84ab8ad82f4d87b2f6109 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c572af9fedc84ab8ad82f4d87b2f6109" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 41399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:10.694399 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.692363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.10.109.66:41399)
0620 18:38:10.692397 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.692398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.692420 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:38:10.692481 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.692482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.692494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.692551 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:10.692663 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.693080 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.693788 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.693960 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.694012 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.694150 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:10.694226 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:10.694295 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.694347 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.694355 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 18:38:10.696012 28884 raft_consensus.cc:1261] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 1 FOLLOWER]: Refusing update from remote peer c572af9fedc84ab8ad82f4d87b2f6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:10.696012 29152 raft_consensus.cc:1261] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 1 FOLLOWER]: Refusing update from remote peer c572af9fedc84ab8ad82f4d87b2f6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:10.696339 29219 consensus_queue.cc:1035] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec862ed1661d4fd78bf679c9adc5fd1f" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:10.696491 29222 consensus_queue.cc:1035] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2f8a7a58574be180eb568e3c630910" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:10.698459 29225 mvcc.cc:205] Tried to move back new op lower bound from 7298992909088342016 to 7298992909017513984. Current Snapshot: MvccSnapshot[applied={T|T < 7298992909088342016}]
I20260620 18:38:10.698902 29227 mvcc.cc:205] Tried to move back new op lower bound from 7298992909088342016 to 7298992909017513984. Current Snapshot: MvccSnapshot[applied={T|T < 7298992909088342016}]
I20260620 18:38:10.699340 29226 mvcc.cc:205] Tried to move back new op lower bound from 7298992909088342016 to 7298992909017513984. Current Snapshot: MvccSnapshot[applied={T|T < 7298992909088342016 or (T in {7298992909088342016})}]
I20260620 18:38:10.700991 28739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:10.701073 28739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:10.701762 28739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5045cce3e443baba6be5ecfb1b0b46: Sending DeleteTablet for 3 replicas of tablet 0baf236b4b284d39ad237b2f126bcecb
I20260620 18:38:10.702098 28864 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf236b4b284d39ad237b2f126bcecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:10 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:58772
I20260620 18:38:10.702103 28998 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf236b4b284d39ad237b2f126bcecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:10 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:51402
I20260620 18:38:10.702317 29235 tablet_replica.cc:330] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: stopping tablet replica
I20260620 18:38:10.702409 29235 raft_consensus.cc:2229] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:10.702451 29132 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf236b4b284d39ad237b2f126bcecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:10 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:36052
I20260620 18:38:10.702672 29235 raft_consensus.cc:2258] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:10.702673 29236 tablet_replica.cc:330] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910: stopping tablet replica
I20260620 18:38:10.702770 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28801
I20260620 18:38:10.702842 29236 raft_consensus.cc:2229] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:10.702936 29236 raft_consensus.cc:2258] T 0baf236b4b284d39ad237b2f126bcecb P 4e2f8a7a58574be180eb568e3c630910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:10.703292 29235 ts_tablet_manager.cc:1905] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:10.703354 29237 tablet_replica.cc:330] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: stopping tablet replica
I20260620 18:38:10.703431 29237 raft_consensus.cc:2229] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:10.703516 29237 raft_consensus.cc:2258] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:10.703974 29237 ts_tablet_manager.cc:1905] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:10.704910 29235 ts_tablet_manager.cc:1918] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:10.704991 29235 log.cc:1196] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/0baf236b4b284d39ad237b2f126bcecb
I20260620 18:38:10.705272 29235 ts_tablet_manager.cc:1939] T 0baf236b4b284d39ad237b2f126bcecb P c572af9fedc84ab8ad82f4d87b2f6109: Deleting consensus metadata
I20260620 18:38:10.705585 29237 ts_tablet_manager.cc:1918] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:10.705648 29237 log.cc:1196] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/0baf236b4b284d39ad237b2f126bcecb
I20260620 18:38:10.705830 29237 ts_tablet_manager.cc:1939] T 0baf236b4b284d39ad237b2f126bcecb P ec862ed1661d4fd78bf679c9adc5fd1f: Deleting consensus metadata
I20260620 18:38:10.706264 28723 catalog_manager.cc:4841] TS ec862ed1661d4fd78bf679c9adc5fd1f (127.10.109.67:45511): tablet 0baf236b4b284d39ad237b2f126bcecb (table test-table [id=fe3dbf98b375427188487b18412da4f1]) successfully deleted
I20260620 18:38:10.707360 28726 catalog_manager.cc:4841] TS c572af9fedc84ab8ad82f4d87b2f6109 (127.10.109.66:41399): tablet 0baf236b4b284d39ad237b2f126bcecb (table test-table [id=fe3dbf98b375427188487b18412da4f1]) successfully deleted
W20260620 18:38:10.710408 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:45847 (error 108)
W20260620 18:38:10.710922 29207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:10.710712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:45847 (local address 127.0.0.1:58784)
0620 18:38:10.710783 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.710794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.710837 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.710873 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:45847: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:38:10.711023 28725 catalog_manager.cc:4584] TS 4e2f8a7a58574be180eb568e3c630910 (127.10.109.65:45847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0baf236b4b284d39ad237b2f126bcecb: Network error: Client connection negotiation failed: client connection to 127.10.109.65:45847: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 18:38:10.711274 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28935
I20260620 18:38:10.716032 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29069
I20260620 18:38:10.720595 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 28707
2026-06-20T18:38:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1915 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:10Z Disabled control of system clock
I20260620 18:38:10.748685 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34109
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:10.826519 29242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.826723 29242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.826758 29242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.827898 29242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:10.827937 29242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.827950 29242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.827962 29242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:10.827975 29242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:10.829288 29242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:34109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.829468 29242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.829757 29242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.834722 29247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.834745 29248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.834908 29242 server_base.cc:1034] running on GCE node
W20260620 18:38:10.834769 29250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.835181 29242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.835480 29242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.836652 29242 hybrid_clock.cc:648] HybridClock initialized: now 1781980690836635 us; error 26 us; skew 500 ppm
I20260620 18:38:10.837862 29242 webserver.cc:460] Webserver started at http://127.10.109.126:40719/ using document root <none> and password file <none>
I20260620 18:38:10.838109 29242 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.838177 29242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.838335 29242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.839428 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "f826effb7e1f4faaa7231bf670703e08"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "50ab5d5e3ebe7164cb358e892eba15a3"
server_key_iv: "d4ac0ad8c381c9826997dbb799173dbf"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.839780 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "f826effb7e1f4faaa7231bf670703e08"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "50ab5d5e3ebe7164cb358e892eba15a3"
server_key_iv: "d4ac0ad8c381c9826997dbb799173dbf"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.841150 29242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:10.841909 29256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.842132 29242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:10.842231 29242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "f826effb7e1f4faaa7231bf670703e08"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "50ab5d5e3ebe7164cb358e892eba15a3"
server_key_iv: "d4ac0ad8c381c9826997dbb799173dbf"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.842312 29242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.858999 29242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.859238 29242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.859335 29242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.863188 29242 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:34109
I20260620 18:38:10.863221 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:34109 every 8 connection(s)
I20260620 18:38:10.863559 29242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:10.864511 29309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:10.866676 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: Bootstrap starting.
I20260620 18:38:10.867270 29309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:10.867573 29309 log.cc:824] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:10.868248 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: No bootstrap required, opened a new log
I20260620 18:38:10.869796 29309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } }
I20260620 18:38:10.869926 29309 raft_consensus.cc:374] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:10.869948 29309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f826effb7e1f4faaa7231bf670703e08, State: Initialized, Role: FOLLOWER
I20260620 18:38:10.870097 29309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } }
I20260620 18:38:10.870165 29309 raft_consensus.cc:388] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:10.870185 29309 raft_consensus.cc:482] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:10.870255 29309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:10.870805 29309 raft_consensus.cc:504] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } }
I20260620 18:38:10.870920 29309 leader_election.cc:302] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f826effb7e1f4faaa7231bf670703e08; no voters:
I20260620 18:38:10.871079 29309 leader_election.cc:288] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:10.871127 29312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:10.871337 29312 raft_consensus.cc:686] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [term 1 LEADER]: Becoming Leader. State: Replica: f826effb7e1f4faaa7231bf670703e08, State: Running, Role: LEADER
I20260620 18:38:10.871352 29309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:10.871446 29312 consensus_queue.cc:237] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } }
I20260620 18:38:10.871901 29314 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f826effb7e1f4faaa7231bf670703e08. Latest consensus state: current_term: 1 leader_uuid: "f826effb7e1f4faaa7231bf670703e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } } }
I20260620 18:38:10.871989 29314 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:10.872324 29313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f826effb7e1f4faaa7231bf670703e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826effb7e1f4faaa7231bf670703e08" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 34109 } } }
I20260620 18:38:10.872393 29313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:10.872509 29321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:10.872889 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29242
I20260620 18:38:10.873003 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:10.873004 29321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:10.873193 10677 external_mini_cluster.cc:1428] Setting key 7a81777414945b4ee11fa4a304903f89
I20260620 18:38:10.874450 29321 catalog_manager.cc:1347] Generated new cluster ID: 96899f7cbfbb44e79a742c1f16458598
I20260620 18:38:10.874502 29321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:10.879433 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.873669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.10.109.126:34109)
0620 18:38:10.873862 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.873870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.873935 (+ 65us) server_negotiation.cc:407] Connection header received
0620 18:38:10.874265 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.874271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.874322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.874466 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.874813 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.876145 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.877162 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.877793 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.877923 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.878606 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.878610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.878610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:10.878855 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.878897 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:10.878911 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.878983 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.879047 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.879225 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260620 18:38:10.887549 29321 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:10.888051 29321 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:10.902945 29321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: Generated new TSK 0
I20260620 18:38:10.903149 29321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:10.904747 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--builtin_ntp_servers=127.10.109.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:10.985152 29333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.985386 29333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.985423 29333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.986539 29333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.986605 29333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.986646 29333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:10.987991 29333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:10.988267 29333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.988560 29333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.994480 29339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.994624 29341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.994493 29338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.994695 29333 server_base.cc:1034] running on GCE node
I20260620 18:38:10.994961 29333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.995209 29333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.996383 29333 hybrid_clock.cc:648] HybridClock initialized: now 1781980690996368 us; error 28 us; skew 500 ppm
I20260620 18:38:10.997706 29333 webserver.cc:460] Webserver started at http://127.10.109.65:35265/ using document root <none> and password file <none>
I20260620 18:38:10.998000 29333 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.998071 29333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.998255 29333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.999354 29333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "eb2c0f84448a41d690d464bdf3bbfdec"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fae7779c9d0ecff63c05fe2df239dbf9"
server_key_iv: "0cbd6be812831ca5891e62df64ed0070"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.999692 29333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "eb2c0f84448a41d690d464bdf3bbfdec"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fae7779c9d0ecff63c05fe2df239dbf9"
server_key_iv: "0cbd6be812831ca5891e62df64ed0070"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.001024 29333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:11.001755 29347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.001924 29333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:11.002024 29333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "eb2c0f84448a41d690d464bdf3bbfdec"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-3jql"
server_key: "fae7779c9d0ecff63c05fe2df239dbf9"
server_key_iv: "0cbd6be812831ca5891e62df64ed0070"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.002102 29333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:11.027699 29333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:11.028017 29333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:11.028177 29333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:11.028414 29333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:11.028743 29333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:11.028800 29333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.028841 29333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:11.028868 29333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.034668 29333 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:41567
I20260620 18:38:11.034788 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:41567 every 8 connection(s)
I20260620 18:38:11.035094 29333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:11.039808 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29333
I20260620 18:38:11.039927 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:11.040153 10677 external_mini_cluster.cc:1428] Setting key d0cd5db6b724e5dc162fd407d813f1d3
I20260620 18:38:11.040169 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.035834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:52119 (local address 127.10.109.126:34109)
0620 18:38:11.035886 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.035888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.036110 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:38:11.036581 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.036583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.036597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.036658 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.037290 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.037792 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.038724 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.038997 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.039064 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.039833 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.039835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.039835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:11.039840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.039859 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:11.039864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.039932 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.040053 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.040091 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 18:38:11.040380 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.035815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:34109 (local address 127.10.109.65:52119)
0620 18:38:11.035986 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.036019 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.036479 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.036713 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.036716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.036723 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.037198 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.037202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.037875 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.037875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.038613 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.038615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.039078 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.039079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.039188 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.039701 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:11.039702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.039727 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:11.039747 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.039974 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.039980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.039983 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.040051 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.040153 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 18:38:11.040901 29461 heartbeater.cc:347] Connected to a master server at 127.10.109.126:34109
I20260620 18:38:11.041000 29461 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:11.041168 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--builtin_ntp_servers=127.10.109.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:38:11.041239 29461 heartbeater.cc:511] Master 127.10.109.126:34109 requested a full tablet report, sending...
I20260620 18:38:11.041721 29273 ts_manager.cc:194] Registered new tserver with Master: eb2c0f84448a41d690d464bdf3bbfdec (127.10.109.65:41567)
I20260620 18:38:11.042536 29273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.65:52119
W20260620 18:38:11.120210 29464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:11.120385 29464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:11.120405 29464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:11.121517 29464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:11.121555 29464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:11.121579 29464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:11.122888 29464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:11.123103 29464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:11.123390 29464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:11.128523 29470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:11.128559 29472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:11.128556 29469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:11.128805 29464 server_base.cc:1034] running on GCE node
I20260620 18:38:11.128997 29464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:11.129230 29464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:11.130403 29464 hybrid_clock.cc:648] HybridClock initialized: now 1781980691130387 us; error 27 us; skew 500 ppm
I20260620 18:38:11.131611 29464 webserver.cc:460] Webserver started at http://127.10.109.66:35627/ using document root <none> and password file <none>
I20260620 18:38:11.131853 29464 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:11.131922 29464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:11.132050 29464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:11.133033 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "f6f936b29ff649afab4c62988fbe668d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "be4143de42d4853caf712527b3c752f9"
server_key_iv: "af28ec31441b3ee6adb26b149798cc2b"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.133378 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "f6f936b29ff649afab4c62988fbe668d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "be4143de42d4853caf712527b3c752f9"
server_key_iv: "af28ec31441b3ee6adb26b149798cc2b"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.134785 29464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:11.135564 29478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.135735 29464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:11.135831 29464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "f6f936b29ff649afab4c62988fbe668d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "be4143de42d4853caf712527b3c752f9"
server_key_iv: "af28ec31441b3ee6adb26b149798cc2b"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.135913 29464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:11.143932 29464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:11.144186 29464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:11.144328 29464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:11.144531 29464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:11.144863 29464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:11.144925 29464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.144964 29464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:11.144991 29464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.150378 29464 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:40335
I20260620 18:38:11.150439 29591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:40335 every 8 connection(s)
I20260620 18:38:11.150804 29464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:11.155766 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.151478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:48789 (local address 127.10.109.126:34109)
0620 18:38:11.151540 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.151542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.151757 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:38:11.152268 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.152269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.152281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.152336 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.152987 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.153487 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.154423 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.154624 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.154705 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.155458 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.155459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.155459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:11.155463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.155478 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:11.155482 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.155549 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.155648 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.155690 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:38:11.155854 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29464
I20260620 18:38:11.155956 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:11.155957 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.151460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:34109 (local address 127.10.109.66:48789)
0620 18:38:11.151647 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.151676 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.152129 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.152398 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.152403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.152416 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.152887 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.152891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.153578 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.153579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.154332 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.154334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.154716 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.154717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.154874 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.155318 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:11.155318 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.155342 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:11.155362 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.155549 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.155554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.155557 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.155636 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.155745 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260620 18:38:11.156165 10677 external_mini_cluster.cc:1428] Setting key 946b69f468feaf16855b0f0d99ed78d3
I20260620 18:38:11.156541 29592 heartbeater.cc:347] Connected to a master server at 127.10.109.126:34109
I20260620 18:38:11.156626 29592 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:11.156828 29592 heartbeater.cc:511] Master 127.10.109.126:34109 requested a full tablet report, sending...
I20260620 18:38:11.157130 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--builtin_ntp_servers=127.10.109.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:38:11.157289 29273 ts_manager.cc:194] Registered new tserver with Master: f6f936b29ff649afab4c62988fbe668d (127.10.109.66:40335)
I20260620 18:38:11.157642 29273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.66:48789
W20260620 18:38:11.235131 29595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:11.235320 29595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:11.235342 29595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:11.236383 29595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:11.236413 29595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:11.236436 29595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:11.237696 29595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:11.237872 29595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:11.238158 29595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:11.243548 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:11.243619 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:11.243662 29595 server_base.cc:1034] running on GCE node
W20260620 18:38:11.243813 29601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:11.244303 29595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:11.244624 29595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:11.245808 29595 hybrid_clock.cc:648] HybridClock initialized: now 1781980691245799 us; error 35 us; skew 500 ppm
I20260620 18:38:11.253250 29595 webserver.cc:460] Webserver started at http://127.10.109.67:34289/ using document root <none> and password file <none>
I20260620 18:38:11.253502 29595 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:11.253573 29595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:11.253727 29595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:11.254860 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "5371ea62d96943d0b350e378fbbbf91d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "1f0038287a33d9e922520b56cdca327a"
server_key_iv: "086a984ad7ba4084b9b46459e634126c"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.255216 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "5371ea62d96943d0b350e378fbbbf91d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "1f0038287a33d9e922520b56cdca327a"
server_key_iv: "086a984ad7ba4084b9b46459e634126c"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.256469 29595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:11.257359 29609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.257586 29595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:11.257687 29595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "5371ea62d96943d0b350e378fbbbf91d"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-3jql"
server_key: "1f0038287a33d9e922520b56cdca327a"
server_key_iv: "086a984ad7ba4084b9b46459e634126c"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.257771 29595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:11.277760 29595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:11.278028 29595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:11.278182 29595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:11.278393 29595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:11.278703 29595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:11.278775 29595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.278815 29595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:11.278843 29595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.284641 29595 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:43489
I20260620 18:38:11.284682 29722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:43489 every 8 connection(s)
I20260620 18:38:11.285044 29595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:11.290401 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.285858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:33029 (local address 127.10.109.126:34109)
0620 18:38:11.285922 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.285924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.286131 (+ 207us) server_negotiation.cc:407] Connection header received
0620 18:38:11.286631 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.286632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.286644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.286702 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.287435 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.287975 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.289031 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.289233 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.289292 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.290111 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.290112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.290112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:11.290116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.290134 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:11.290138 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.290185 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.290311 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.290336 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:38:11.290495 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.285801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:34109 (local address 127.10.109.67:33029)
0620 18:38:11.286012 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.286042 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.286486 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.286840 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.286850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.286858 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.287335 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.287339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.288130 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.288131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.288937 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.288940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.289322 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.289323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.289425 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.289978 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:11.289978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.290005 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:11.290025 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.290185 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.290187 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.290189 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.290238 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.290307 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260620 18:38:11.291074 29723 heartbeater.cc:347] Connected to a master server at 127.10.109.126:34109
I20260620 18:38:11.291174 29723 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:11.291374 29723 heartbeater.cc:511] Master 127.10.109.126:34109 requested a full tablet report, sending...
I20260620 18:38:11.291767 29273 ts_manager.cc:194] Registered new tserver with Master: 5371ea62d96943d0b350e378fbbbf91d (127.10.109.67:43489)
I20260620 18:38:11.291888 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29595
I20260620 18:38:11.292012 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:11.292220 10677 external_mini_cluster.cc:1428] Setting key 352a12025019f3c30878217ce7e01850
I20260620 18:38:11.292213 29273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.67:33029
I20260620 18:38:11.293104 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:11.297335 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.294428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.10.109.126:34109)
0620 18:38:11.294505 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.294507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.294627 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:38:11.294692 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.294693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.294704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.294792 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.294984 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.295455 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.296193 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.296382 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.296464 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.297057 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.297059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.297059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:11.297063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.297074 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:11.297078 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.297179 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.297252 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.297262 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:11.298805 29273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:11.300356 29273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:11.306770 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.10.109.66:40335)
0620 18:38:11.303116 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.303120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.303130 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:11.303170 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.303172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.303186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.303268 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.303359 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304514 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.305773 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306351 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.306438 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.306518 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.306630 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.306665 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260620 18:38:11.306766 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40335 (local address 127.0.0.1:58316)
0620 18:38:11.302952 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.302961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.303007 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.303262 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.303263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.303265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.303300 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.303301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304608 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.304610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.305675 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.305678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306450 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.306451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.306532 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.306546 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.306616 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.306665 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260620 18:38:11.306910 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41567 (local address 127.0.0.1:46184)
0620 18:38:11.302675 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.302711 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.302806 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.303298 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.303299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.303302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.303347 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.303350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304654 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.304655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.305786 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.305788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306668 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.306669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.306734 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.306757 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.306788 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.306823 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260620 18:38:11.307112 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.10.109.67:43489)
0620 18:38:11.302884 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.302888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.302896 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:11.302925 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.302927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.302949 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.303034 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.303184 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304165 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.305613 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306368 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.306437 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.306516 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.306806 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.307042 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":95,"threads_started":1}
I20260620 18:38:11.307255 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:43489 (local address 127.0.0.1:33726)
0620 18:38:11.302672 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.302719 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.302805 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.303038 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.303040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.303043 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.303100 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.303102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304265 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.304266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.305549 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.305552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306553 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.306554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.306624 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.306630 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.306677 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.306716 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":45,"threads_started":1}
I20260620 18:38:11.307341 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.302657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.10.109.65:41567)
0620 18:38:11.303124 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.303128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.303137 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:11.303173 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.303174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.303188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.303424 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.303487 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.304573 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.305871 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.306579 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.307078 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.307181 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.307211 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.307253 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":111,"threads_started":1}
I20260620 18:38:11.307436 29526 tablet_service.cc:1467] Processing CreateTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d (DEFAULT_TABLE table=test-table [id=fd5d2649ef8f41d8a6af59268023ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.307776 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.307922 29395 tablet_service.cc:1467] Processing CreateTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d (DEFAULT_TABLE table=test-table [id=fd5d2649ef8f41d8a6af59268023ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.308408 29395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.311036 29737 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Bootstrap starting.
I20260620 18:38:11.311058 29657 tablet_service.cc:1467] Processing CreateTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d (DEFAULT_TABLE table=test-table [id=fd5d2649ef8f41d8a6af59268023ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.311374 29657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.311707 29737 tablet_bootstrap.cc:654] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.312026 29737 log.cc:824] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.312611 29739 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Bootstrap starting.
I20260620 18:38:11.312775 29737 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: No bootstrap required, opened a new log
I20260620 18:38:11.312849 29737 ts_tablet_manager.cc:1397] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:11.313380 29739 tablet_bootstrap.cc:654] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.313755 29739 log.cc:824] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.314385 29741 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Bootstrap starting.
I20260620 18:38:11.314908 29739 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: No bootstrap required, opened a new log
I20260620 18:38:11.314989 29739 ts_tablet_manager.cc:1397] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:11.315012 29741 tablet_bootstrap.cc:654] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.315306 29741 log.cc:824] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.315676 29737 raft_consensus.cc:348] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.315855 29737 raft_consensus.cc:374] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.315897 29737 raft_consensus.cc:729] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2c0f84448a41d690d464bdf3bbfdec, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.316054 29741 tablet_bootstrap.cc:492] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: No bootstrap required, opened a new log
I20260620 18:38:11.316035 29737 consensus_queue.cc:260] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.316114 29741 ts_tablet_manager.cc:1397] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:11.316335 29461 heartbeater.cc:503] Master 127.10.109.126:34109 was elected leader, sending a full tablet report...
I20260620 18:38:11.317143 29737 ts_tablet_manager.cc:1428] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:38:11.317109 29739 raft_consensus.cc:348] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.317252 29739 raft_consensus.cc:374] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.317291 29739 raft_consensus.cc:729] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f936b29ff649afab4c62988fbe668d, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.317405 29739 consensus_queue.cc:260] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.317679 29739 ts_tablet_manager.cc:1428] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:11.317723 29592 heartbeater.cc:503] Master 127.10.109.126:34109 was elected leader, sending a full tablet report...
I20260620 18:38:11.318663 29741 raft_consensus.cc:348] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.318843 29741 raft_consensus.cc:374] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.318868 29741 raft_consensus.cc:729] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5371ea62d96943d0b350e378fbbbf91d, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.318936 29741 consensus_queue.cc:260] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.319108 29741 ts_tablet_manager.cc:1428] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:11.319175 29723 heartbeater.cc:503] Master 127.10.109.126:34109 was elected leader, sending a full tablet report...
W20260620 18:38:11.401383 29593 tablet.cc:2369] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:11.517760 29745 raft_consensus.cc:482] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:11.517882 29745 raft_consensus.cc:504] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.518357 29745 leader_election.cc:288] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2c0f84448a41d690d464bdf3bbfdec (127.10.109.65:41567), f6f936b29ff649afab4c62988fbe668d (127.10.109.66:40335)
I20260620 18:38:11.521265 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.518625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41567 (local address 127.10.109.67:57867)
0620 18:38:11.518710 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.518724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.518811 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.518994 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.518995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.518997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.519064 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.519066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.519724 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.519725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.520469 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.520471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.521011 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.521013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.521078 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.521081 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.521154 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.521194 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:38:11.521265 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.518716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:57867 (local address 127.10.109.65:41567)
0620 18:38:11.518804 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.518807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.518820 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:38:11.518916 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.518918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.518930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.518982 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.519151 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.519627 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.520574 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.520919 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.521000 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.521008 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.521162 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.521198 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:38:11.521566 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.518865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:40335 (local address 127.10.109.67:45081)
0620 18:38:11.519027 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.519044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.519111 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.519230 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.519230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.519232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.519302 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.519304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.519921 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.519922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.520645 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.520646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.521175 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.521176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.521251 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.521254 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.521460 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.521501 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260620 18:38:11.521598 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.518871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:45081 (local address 127.10.109.66:40335)
0620 18:38:11.518916 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.518918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.519102 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:38:11.519173 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.519174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.519182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.519229 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.519372 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.519836 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.520740 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.521083 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.521181 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.521189 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.521490 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.521539 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:38:11.521605 29415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b68600ee59c4e8d99d47cdb3cc0df3d" candidate_uuid: "5371ea62d96943d0b350e378fbbbf91d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" is_pre_election: true
I20260620 18:38:11.521754 29415 raft_consensus.cc:2454] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5371ea62d96943d0b350e378fbbbf91d in term 0.
I20260620 18:38:11.521942 29546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b68600ee59c4e8d99d47cdb3cc0df3d" candidate_uuid: "5371ea62d96943d0b350e378fbbbf91d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f936b29ff649afab4c62988fbe668d" is_pre_election: true
I20260620 18:38:11.521981 29613 leader_election.cc:302] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5371ea62d96943d0b350e378fbbbf91d, eb2c0f84448a41d690d464bdf3bbfdec; no voters:
I20260620 18:38:11.522056 29546 raft_consensus.cc:2454] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5371ea62d96943d0b350e378fbbbf91d in term 0.
I20260620 18:38:11.522091 29745 raft_consensus.cc:2781] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:11.522150 29745 raft_consensus.cc:482] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:11.522197 29745 raft_consensus.cc:3037] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.522919 29745 raft_consensus.cc:504] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.523021 29745 leader_election.cc:288] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [CANDIDATE]: Term 1 election: Requested vote from peers eb2c0f84448a41d690d464bdf3bbfdec (127.10.109.65:41567), f6f936b29ff649afab4c62988fbe668d (127.10.109.66:40335)
I20260620 18:38:11.523236 29415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b68600ee59c4e8d99d47cdb3cc0df3d" candidate_uuid: "5371ea62d96943d0b350e378fbbbf91d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2c0f84448a41d690d464bdf3bbfdec"
I20260620 18:38:11.523278 29546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b68600ee59c4e8d99d47cdb3cc0df3d" candidate_uuid: "5371ea62d96943d0b350e378fbbbf91d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f936b29ff649afab4c62988fbe668d"
I20260620 18:38:11.523329 29546 raft_consensus.cc:3037] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.523332 29415 raft_consensus.cc:3037] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.523970 29546 raft_consensus.cc:2454] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5371ea62d96943d0b350e378fbbbf91d in term 1.
I20260620 18:38:11.524137 29613 leader_election.cc:302] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5371ea62d96943d0b350e378fbbbf91d, f6f936b29ff649afab4c62988fbe668d; no voters:
I20260620 18:38:11.524184 29415 raft_consensus.cc:2454] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5371ea62d96943d0b350e378fbbbf91d in term 1.
I20260620 18:38:11.524266 29745 raft_consensus.cc:2781] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:11.524406 29745 raft_consensus.cc:686] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 1 LEADER]: Becoming Leader. State: Replica: 5371ea62d96943d0b350e378fbbbf91d, State: Running, Role: LEADER
I20260620 18:38:11.524507 29745 consensus_queue.cc:237] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } }
I20260620 18:38:11.525276 29273 catalog_manager.cc:5495] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5371ea62d96943d0b350e378fbbbf91d (127.10.109.67). New cstate: current_term: 1 leader_uuid: "5371ea62d96943d0b350e378fbbbf91d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5371ea62d96943d0b350e378fbbbf91d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:11.532300 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.530039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.10.109.67:43489)
0620 18:38:11.530091 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.530093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.530100 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:38:11.530128 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.530132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.530146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.530208 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:11.530312 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.530725 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.531525 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.531732 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.531803 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.531976 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.532062 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.532165 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.532204 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.532212 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:11.533792 29546 raft_consensus.cc:1261] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 1 FOLLOWER]: Refusing update from remote peer 5371ea62d96943d0b350e378fbbbf91d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:11.533847 29415 raft_consensus.cc:1261] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 1 FOLLOWER]: Refusing update from remote peer 5371ea62d96943d0b350e378fbbbf91d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:11.534090 29745 consensus_queue.cc:1035] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f936b29ff649afab4c62988fbe668d" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:11.534252 29747 consensus_queue.cc:1035] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2c0f84448a41d690d464bdf3bbfdec" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:38:11.535674 29724 tablet.cc:2369] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:11.535971 29462 tablet.cc:2369] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:11.537604 29751 mvcc.cc:205] Tried to move back new op lower bound from 7298992912520323072 to 7298992912485044224. Current Snapshot: MvccSnapshot[applied={T|T < 7298992912520323072}]
I20260620 18:38:11.538926 29273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:11.539013 29273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:11.539726 29273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f826effb7e1f4faaa7231bf670703e08: Sending DeleteTablet for 3 replicas of tablet 1b68600ee59c4e8d99d47cdb3cc0df3d
I20260620 18:38:11.540011 29657 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:11 UTC) from {username='slave'} at 127.0.0.1:33726
I20260620 18:38:11.540050 29526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:11 UTC) from {username='slave'} at 127.0.0.1:58316
I20260620 18:38:11.540161 29395 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:11 UTC) from {username='slave'} at 127.0.0.1:46184
I20260620 18:38:11.540306 29761 tablet_replica.cc:330] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: stopping tablet replica
I20260620 18:38:11.540355 29762 tablet_replica.cc:330] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: stopping tablet replica
I20260620 18:38:11.540452 29762 raft_consensus.cc:2229] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:11.540478 29760 tablet_replica.cc:330] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: stopping tablet replica
I20260620 18:38:11.540548 29762 raft_consensus.cc:2258] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:11.540647 29761 raft_consensus.cc:2229] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:11.540746 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29333
I20260620 18:38:11.540997 29762 ts_tablet_manager.cc:1905] T 1b68600ee59c4e8d99d47cdb3cc0df3d P eb2c0f84448a41d690d464bdf3bbfdec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:11.540796 29761 raft_consensus.cc:2258] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:11.542070 29761 ts_tablet_manager.cc:1905] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:11.543677 29761 ts_tablet_manager.cc:1918] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:11.543809 29761 log.cc:1196] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/1b68600ee59c4e8d99d47cdb3cc0df3d
I20260620 18:38:11.544127 29761 ts_tablet_manager.cc:1939] T 1b68600ee59c4e8d99d47cdb3cc0df3d P f6f936b29ff649afab4c62988fbe668d: Deleting consensus metadata
I20260620 18:38:11.544525 29260 catalog_manager.cc:4841] TS f6f936b29ff649afab4c62988fbe668d (127.10.109.66:40335): tablet 1b68600ee59c4e8d99d47cdb3cc0df3d (table test-table [id=fd5d2649ef8f41d8a6af59268023ca6a]) successfully deleted
W20260620 18:38:11.546958 29260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:41567 (error 108)
I20260620 18:38:11.547148 29760 raft_consensus.cc:2229] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:11.547201 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29464
I20260620 18:38:11.547255 29760 raft_consensus.cc:2258] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:11.548406 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.548264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:41567 (local address 127.0.0.1:46194)
0620 18:38:11.548322 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.548370 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:38:11.548588 29260 catalog_manager.cc:4584] TS eb2c0f84448a41d690d464bdf3bbfdec (127.10.109.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b68600ee59c4e8d99d47cdb3cc0df3d: Network error: Client connection negotiation failed: client connection to 127.10.109.65:41567: connect: Connection refused (error 111)
I20260620 18:38:11.551149 29760 ts_tablet_manager.cc:1905] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:11.552886 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29595
I20260620 18:38:11.553046 29760 ts_tablet_manager.cc:1918] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:11.553108 29760 log.cc:1196] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/1b68600ee59c4e8d99d47cdb3cc0df3d
I20260620 18:38:11.553316 29760 ts_tablet_manager.cc:1939] T 1b68600ee59c4e8d99d47cdb3cc0df3d P 5371ea62d96943d0b350e378fbbbf91d: Deleting consensus metadata
I20260620 18:38:11.556991 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29242
I20260620 18:38:11.557089 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.556976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:43489 (local address 127.0.0.1:33732)
0620 18:38:11.557030 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.557054 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.67:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
2026-06-20T18:38:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:11 dist-test-slave-3jql krb5kdc[29766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:11 dist-test-slave-3jql krb5kdc[29766](info): set up 2 sockets
Jun 20 18:38:11 dist-test-slave-3jql krb5kdc[29766](info): commencing operation
krb5kdc: starting...
W20260620 18:38:12.590631 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:12 dist-test-slave-3jql krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:12.707549 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45619
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:12.787937 29779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:12.788125 29779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:12.788146 29779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:12.789203 29779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:12.789235 29779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:12.789250 29779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:12.789263 29779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:12.789274 29779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:12.790622 29779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:12.790869 29779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:12.791043 29779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:12.795986 29785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:12.796059 29779 server_base.cc:1034] running on GCE node
W20260620 18:38:12.795975 29784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:12.795975 29787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:12.796415 29779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:12.796698 29779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:12.797842 29779 hybrid_clock.cc:648] HybridClock initialized: now 1781980692797820 us; error 36 us; skew 500 ppm
Jun 20 18:38:12 dist-test-slave-3jql krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980692, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:12.800273 29779 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:12.800719 29779 webserver.cc:460] Webserver started at http://127.10.109.126:35219/ using document root <none> and password file <none>
I20260620 18:38:12.800935 29779 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:12.801002 29779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:12.801148 29779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:12.802222 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "4797046f09924a3a81049500c06ed2e9"
format_stamp: "Formatted at 2026-06-20 18:38:12 on dist-test-slave-3jql"
server_key: "397f8d95eca432008ab6a214f2deee1b"
server_key_iv: "e0d62d9d7708f9c4e4ec6a5fecca0791"
server_key_version: "encryptionkey@0"
I20260620 18:38:12.802561 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "4797046f09924a3a81049500c06ed2e9"
format_stamp: "Formatted at 2026-06-20 18:38:12 on dist-test-slave-3jql"
server_key: "397f8d95eca432008ab6a214f2deee1b"
server_key_iv: "e0d62d9d7708f9c4e4ec6a5fecca0791"
server_key_version: "encryptionkey@0"
I20260620 18:38:12.803936 29779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:12.804720 29794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:12.804955 29779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:12.805073 29779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "4797046f09924a3a81049500c06ed2e9"
format_stamp: "Formatted at 2026-06-20 18:38:12 on dist-test-slave-3jql"
server_key: "397f8d95eca432008ab6a214f2deee1b"
server_key_iv: "e0d62d9d7708f9c4e4ec6a5fecca0791"
server_key_version: "encryptionkey@0"
I20260620 18:38:12.805153 29779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:12.833459 29779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:12.835974 29779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:12.836143 29779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:12.840322 29779 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:45619
I20260620 18:38:12.840370 29846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:45619 every 8 connection(s)
I20260620 18:38:12.840736 29779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:12.841540 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:12.842185 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29779
I20260620 18:38:12.842348 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:12.842533 10677 external_mini_cluster.cc:1428] Setting key 1355a7bfc68e182aa09c883ed8f4c431
I20260620 18:38:12.844561 29847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: Bootstrap starting.
I20260620 18:38:12.845268 29847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:12.845573 29847 log.cc:824] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:12.846300 29847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: No bootstrap required, opened a new log
I20260620 18:38:12.848577 29847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } }
I20260620 18:38:12.848711 29847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:12.848731 29847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4797046f09924a3a81049500c06ed2e9, State: Initialized, Role: FOLLOWER
I20260620 18:38:12.848856 29847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } }
I20260620 18:38:12.848933 29847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:12.848982 29847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:12.849032 29847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 18:38:12 dist-test-slave-3jql krb5kdc[29766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:12.849538 29847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } }
I20260620 18:38:12.849668 29847 leader_election.cc:302] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4797046f09924a3a81049500c06ed2e9; no voters:
I20260620 18:38:12.849820 29847 leader_election.cc:288] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:12.849869 29852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:12.850040 29852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4797046f09924a3a81049500c06ed2e9, State: Running, Role: LEADER
I20260620 18:38:12.850076 29847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:12.850150 29852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } }
I20260620 18:38:12.850445 29854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4797046f09924a3a81049500c06ed2e9. Latest consensus state: current_term: 1 leader_uuid: "4797046f09924a3a81049500c06ed2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } } }
I20260620 18:38:12.850528 29854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:12.850728 29853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4797046f09924a3a81049500c06ed2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4797046f09924a3a81049500c06ed2e9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45619 } } }
I20260620 18:38:12.850826 29853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:12.851249 29859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:12.852571 29859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:12.853865 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.843035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.10.109.126:45619)
0620 18:38:12.843245 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:12.843254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:12.843298 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:38:12.843844 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:12.843852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:12.843919 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:12.844062 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:12.844644 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.845965 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.846771 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.847381 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.847495 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.850165 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:12.850170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:12.850171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:12.850181 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:12.852171 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:12.852551 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:12.852554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:12.852554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:12.852666 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:12.852944 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:12.852945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:12.852946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:12.853240 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:12.853390 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:12.853514 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:38:12.853657 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260620 18:38:12.854138 29859 catalog_manager.cc:1347] Generated new cluster ID: 526705bf97f743efb1cdc929cb109c3a
I20260620 18:38:12.854179 29859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:12.864670 29859 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:12.865355 29859 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:12.871563 29859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: Generated new TSK 0
I20260620 18:38:12.871745 29859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.109.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
Entry for principal kudu/127.10.109.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab.
I20260620 18:38:12.905110 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:12.999821 29873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:13.000044 29873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:13.000097 29873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:13.001273 29873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:13.001335 29873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:13.001379 29873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:13.002893 29873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.65.keytab
--principal=kudu/127.10.109.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:13.003119 29873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:13.003490 29873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:13.008463 29881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.008463 29879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.008523 29878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:13.009399 29873 server_base.cc:1034] running on GCE node
I20260620 18:38:13.009976 29873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:13.010285 29873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:13.011487 29873 hybrid_clock.cc:648] HybridClock initialized: now 1781980693011458 us; error 44 us; skew 500 ppm
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:13.014008 29873 init.cc:377] Logged in from keytab as kudu/127.10.109.65@KRBTEST.COM (short username kudu)
I20260620 18:38:13.014364 29873 webserver.cc:460] Webserver started at http://127.10.109.65:40991/ using document root <none> and password file <none>
I20260620 18:38:13.014578 29873 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:13.014623 29873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:13.014811 29873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:13.015812 29873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "1c776262d53a4e129975c240c8afa581"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "08d831a93616d9d4cc7487c75d1b0adb"
server_key_iv: "1730658d6be1bb6da42ea88ae928bfa9"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.016132 29873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "1c776262d53a4e129975c240c8afa581"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "08d831a93616d9d4cc7487c75d1b0adb"
server_key_iv: "1730658d6be1bb6da42ea88ae928bfa9"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.017431 29873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:13.018169 29888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.018383 29873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:13.018460 29873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "1c776262d53a4e129975c240c8afa581"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "08d831a93616d9d4cc7487c75d1b0adb"
server_key_iv: "1730658d6be1bb6da42ea88ae928bfa9"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.018548 29873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:13.031873 29873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:13.033839 29873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:13.033950 29873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:13.034245 29873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:13.034555 29873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:13.034622 29873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.034667 29873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:13.034706 29873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.040397 29873 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:35683
I20260620 18:38:13.040467 30001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:35683 every 8 connection(s)
I20260620 18:38:13.040779 29873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.65@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:13.049496 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.041570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:59463 (local address 127.10.109.126:45619)
0620 18:38:13.041627 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.041630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.041810 (+ 180us) server_negotiation.cc:407] Connection header received
0620 18:38:13.042275 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.042276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.042285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.042346 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.043249 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.043766 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.044700 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.044912 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.044965 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.046951 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.046952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.046953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:13.046958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.048387 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.048726 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.048726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.048727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.048774 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.048984 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.048984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.048985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.049081 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.049143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.049363 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.049433 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:38:13.049611 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.041515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45619 (local address 127.10.109.65:59463)
0620 18:38:13.041686 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.041727 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.042197 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.042396 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.042399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.042977 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:13.043168 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.043173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.043856 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.043857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.044620 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.044622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.044987 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.044988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.045081 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.045491 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:13.045500 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:13.046873 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:13.048452 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.048457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.048459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.048654 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.048831 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.048831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.048832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.048926 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.049145 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:13.049149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:13.049253 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.049333 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.049438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260620 18:38:13.050251 30002 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45619
I20260620 18:38:13.050384 30002 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:13.050458 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 29873
I20260620 18:38:13.050572 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:13.050594 30002 heartbeater.cc:511] Master 127.10.109.126:45619 requested a full tablet report, sending...
I20260620 18:38:13.050827 10677 external_mini_cluster.cc:1428] Setting key 22f21b831c3cf3fee65eaded773120f1
I20260620 18:38:13.051290 29811 ts_manager.cc:194] Registered new tserver with Master: 1c776262d53a4e129975c240c8afa581 (127.10.109.65:35683)
I20260620 18:38:13.052016 29811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.65@KRBTEST.COM'} at 127.10.109.65:59463
WARNING: no policy specified for kudu/127.10.109.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
Entry for principal kudu/127.10.109.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab.
I20260620 18:38:13.077064 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:13.158133 30007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:13.158341 30007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:13.158363 30007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:13.159735 30007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:13.159788 30007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:13.159816 30007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:13.161284 30007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.66.keytab
--principal=kudu/127.10.109.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:13.161450 30007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:13.161612 30007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:13.166289 30015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.166323 30013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.166307 30012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:13.166599 30007 server_base.cc:1034] running on GCE node
I20260620 18:38:13.166814 30007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:13.167043 30007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:13.168231 30007 hybrid_clock.cc:648] HybridClock initialized: now 1781980693168200 us; error 47 us; skew 500 ppm
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:13.170565 30007 init.cc:377] Logged in from keytab as kudu/127.10.109.66@KRBTEST.COM (short username kudu)
I20260620 18:38:13.171017 30007 webserver.cc:460] Webserver started at http://127.10.109.66:43125/ using document root <none> and password file <none>
I20260620 18:38:13.171233 30007 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:13.171298 30007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:13.171437 30007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:13.172405 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "c4a3dd65bc2843ba889ac6fc749550b5"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "4767361c9c002824be4aee2b70240044"
server_key_iv: "90328747474f9ddc5c6829972b0af198"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.172746 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "c4a3dd65bc2843ba889ac6fc749550b5"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "4767361c9c002824be4aee2b70240044"
server_key_iv: "90328747474f9ddc5c6829972b0af198"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.173988 30007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:13.174702 30022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.174906 30007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:13.175005 30007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "c4a3dd65bc2843ba889ac6fc749550b5"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "4767361c9c002824be4aee2b70240044"
server_key_iv: "90328747474f9ddc5c6829972b0af198"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.175082 30007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:13.184598 30007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:13.186370 30007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:13.186514 30007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:13.186721 30007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:13.187034 30007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:13.187084 30007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.187122 30007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:13.187150 30007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.192806 30007 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:39169
I20260620 18:38:13.192839 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:39169 every 8 connection(s)
I20260620 18:38:13.193127 30007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.66@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:13.201469 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30007
I20260620 18:38:13.201596 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:13.201749 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.193793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:41265 (local address 127.10.109.126:45619)
0620 18:38:13.193855 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.193857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.194115 (+ 258us) server_negotiation.cc:407] Connection header received
0620 18:38:13.194570 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.194571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.194581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.194634 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.195517 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.196005 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.196942 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.197119 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.197199 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.199196 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.199198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.199198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:13.199204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.200555 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.200908 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.200908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.200909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.200958 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.201156 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.201156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.201157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.201267 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.201319 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.201558 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.201663 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:13.201799 10677 external_mini_cluster.cc:1428] Setting key 6d4d1c36b62a020e9460c4015a0e2a6e
I20260620 18:38:13.201936 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.193794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45619 (local address 127.10.109.66:41265)
0620 18:38:13.193974 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.194029 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.194484 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.194680 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.194683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.195258 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:13.195429 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.195433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.196090 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.196090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.196871 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.196874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.197281 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.197282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.197389 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.197785 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:13.197792 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:13.199103 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:13.200625 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.200630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.200632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.200843 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.201016 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.201017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.201017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.201103 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.201318 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:13.201322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:13.201418 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.201520 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.201675 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260620 18:38:13.202337 30136 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45619
I20260620 18:38:13.202450 30136 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:13.202620 30136 heartbeater.cc:511] Master 127.10.109.126:45619 requested a full tablet report, sending...
I20260620 18:38:13.203018 29811 ts_manager.cc:194] Registered new tserver with Master: c4a3dd65bc2843ba889ac6fc749550b5 (127.10.109.66:39169)
I20260620 18:38:13.203413 29811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.66@KRBTEST.COM'} at 127.10.109.66:41265
WARNING: no policy specified for kudu/127.10.109.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
Entry for principal kudu/127.10.109.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab.
I20260620 18:38:13.227391 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:13.308135 30141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:13.308348 30141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:13.308388 30141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:13.309593 30141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:13.309653 30141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:13.309701 30141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:13.311254 30141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.67.keytab
--principal=kudu/127.10.109.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:45619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:13.311508 30141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:13.311734 30141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:13.316607 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.316648 30146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.316648 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:13.316908 30141 server_base.cc:1034] running on GCE node
I20260620 18:38:13.317102 30141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:13.317351 30141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:13.318514 30141 hybrid_clock.cc:648] HybridClock initialized: now 1781980693318503 us; error 29 us; skew 500 ppm
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:13.321055 30141 init.cc:377] Logged in from keytab as kudu/127.10.109.67@KRBTEST.COM (short username kudu)
I20260620 18:38:13.321543 30141 webserver.cc:460] Webserver started at http://127.10.109.67:39589/ using document root <none> and password file <none>
I20260620 18:38:13.321789 30141 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:13.321854 30141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:13.321998 30141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:13.323104 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "ccbf4a75d63944d09570642995cabb5d"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "3c0b402b95ea23d2e3d4a9ecfc8f6deb"
server_key_iv: "1247705eeeb0f8e64fc7620f43e32cc1"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.323441 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "ccbf4a75d63944d09570642995cabb5d"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "3c0b402b95ea23d2e3d4a9ecfc8f6deb"
server_key_iv: "1247705eeeb0f8e64fc7620f43e32cc1"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.324728 30141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:13.325467 30156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.325632 30141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:13.325739 30141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "ccbf4a75d63944d09570642995cabb5d"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "3c0b402b95ea23d2e3d4a9ecfc8f6deb"
server_key_iv: "1247705eeeb0f8e64fc7620f43e32cc1"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.325809 30141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:13.339224 30141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:13.341346 30141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:13.341454 30141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:13.341674 30141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:13.341976 30141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:13.342013 30141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.342033 30141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:13.342046 30141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.347864 30141 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:36759
I20260620 18:38:13.347930 30269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:36759 every 8 connection(s)
I20260620 18:38:13.348238 30141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:13.351768 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30141
I20260620 18:38:13.351886 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:13.352047 10677 external_mini_cluster.cc:1428] Setting key 16216a01bfc009f8c9fe83c6d6a547c1
Jun 20 18:38:13 dist-test-slave-3jql krb5kdc[29766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.67@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:13.357344 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.349011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:40141 (local address 127.10.109.126:45619)
0620 18:38:13.349071 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.349073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.349280 (+ 207us) server_negotiation.cc:407] Connection header received
0620 18:38:13.349817 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.349817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.349828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.349943 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.350984 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.351583 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.352958 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.353149 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.353200 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.355089 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.355090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.355090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:13.355096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.356220 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.356566 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.356567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.356567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.356615 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.356817 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.356818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.356818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.356919 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.356973 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.357222 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.357284 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:38:13.357498 30272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.349008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45619 (local address 127.10.109.67:40141)
0620 18:38:13.349165 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.349197 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.349711 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.349993 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.349996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.350615 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:13.350899 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.350904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.351702 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.351704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.352898 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.352901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.353215 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.353215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.353312 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.353701 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:13.353709 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:13.355011 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:13.356290 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.356296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.356297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.356503 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.356672 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:13.356672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:13.356673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:13.356762 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:13.356972 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:13.356975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:13.357075 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.357187 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.357297 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20260620 18:38:13.357945 30270 heartbeater.cc:347] Connected to a master server at 127.10.109.126:45619
I20260620 18:38:13.358050 30270 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:13.358208 30270 heartbeater.cc:511] Master 127.10.109.126:45619 requested a full tablet report, sending...
I20260620 18:38:13.358583 29811 ts_manager.cc:194] Registered new tserver with Master: ccbf4a75d63944d09570642995cabb5d (127.10.109.67:36759)
I20260620 18:38:13.358985 29811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.109.67@KRBTEST.COM'} at 127.10.109.67:40141
I20260620 18:38:13.363615 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:13.371768 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.365156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.10.109.126:45619)
0620 18:38:13.365191 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.365193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.365352 (+ 159us) server_negotiation.cc:407] Connection header received
0620 18:38:13.365384 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.365385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.365400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.365461 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.366154 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.366883 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.367984 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.368294 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.368358 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.369461 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.369462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.369462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:13.369468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.370673 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.371009 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.371010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.371010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.371057 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:13.371186 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:13.371186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:13.371186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:13.371335 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.371411 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.371562 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.371656 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:13.373281 29811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:13.374866 29811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:13.380918 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.376925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:35683 (local address 127.0.0.1:38052)
0620 18:38:13.377052 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.377136 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.377198 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.377584 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.377586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.377588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:13.377635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.377636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.378734 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.378735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.379623 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.379625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.380648 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.380649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.380731 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.380743 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.380798 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.380849 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":39,"threads_started":1}
I20260620 18:38:13.380942 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.377201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.10.109.65:35683)
0620 18:38:13.377366 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.377370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.377381 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:13.377423 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.377425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.377446 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.377580 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:13.377711 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.378640 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.379722 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.380491 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.380537 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.380587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.380829 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.380868 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260620 18:38:13.381420 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.377101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:36759 (local address 127.0.0.1:41624)
0620 18:38:13.377830 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.377843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.377892 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.378107 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.378108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.378110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:13.378153 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.378155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.379465 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.379466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.380359 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.380362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.381201 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.381202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.381270 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.381273 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.381319 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.381355 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":79,"threads_started":1}
I20260620 18:38:13.381717 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.377176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.10.109.67:36759)
0620 18:38:13.377342 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.377346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.377893 (+ 547us) server_negotiation.cc:407] Connection header received
0620 18:38:13.377957 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.377959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.377976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.378064 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:13.378227 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.379341 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.380441 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.381056 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.381122 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.381169 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.381599 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.381641 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260620 18:38:13.381687 29936 tablet_service.cc:1467] Processing CreateTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 (DEFAULT_TABLE table=test-table [id=318613b5e686403bb8f92cd32739fd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:13.382035 29936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b3616b06c74e70893e77d49f2f6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:13.382139 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.376932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:39169 (local address 127.0.0.1:51744)
0620 18:38:13.377656 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.377704 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.377757 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.378160 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.378160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.378162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:13.378273 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.378276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.379532 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.379533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.380654 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.380655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.381872 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.381873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.381935 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.381938 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.381984 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.382022 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260620 18:38:13.382318 30204 tablet_service.cc:1467] Processing CreateTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 (DEFAULT_TABLE table=test-table [id=318613b5e686403bb8f92cd32739fd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:13.382565 30204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b3616b06c74e70893e77d49f2f6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:13.383101 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.377030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.10.109.66:39169)
0620 18:38:13.377366 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.377370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.377959 (+ 589us) server_negotiation.cc:407] Connection header received
0620 18:38:13.378004 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.378006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.378020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.378086 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:13.378355 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.379428 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.380731 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.381694 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.381762 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.382942 (+ 1180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.382972 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.383010 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":68,"threads_started":1}
I20260620 18:38:13.384846 30287 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Bootstrap starting.
I20260620 18:38:13.385102 30288 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Bootstrap starting.
I20260620 18:38:13.385305 30070 tablet_service.cc:1467] Processing CreateTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 (DEFAULT_TABLE table=test-table [id=318613b5e686403bb8f92cd32739fd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:13.385682 30287 tablet_bootstrap.cc:654] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:13.385738 30070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b3616b06c74e70893e77d49f2f6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:13.385869 30288 tablet_bootstrap.cc:654] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:13.386317 30288 log.cc:824] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:13.386335 30287 log.cc:824] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:13.387024 30288 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: No bootstrap required, opened a new log
I20260620 18:38:13.387104 30288 ts_tablet_manager.cc:1397] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:13.388376 30291 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Bootstrap starting.
I20260620 18:38:13.388510 30288 raft_consensus.cc:348] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.388701 30288 raft_consensus.cc:374] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:13.388749 30288 raft_consensus.cc:729] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbf4a75d63944d09570642995cabb5d, State: Initialized, Role: FOLLOWER
I20260620 18:38:13.388861 30288 consensus_queue.cc:260] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.388993 30291 tablet_bootstrap.cc:654] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:13.389145 30288 ts_tablet_manager.cc:1428] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:13.389165 30270 heartbeater.cc:503] Master 127.10.109.126:45619 was elected leader, sending a full tablet report...
I20260620 18:38:13.389518 30291 log.cc:824] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:13.389544 30287 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: No bootstrap required, opened a new log
I20260620 18:38:13.389602 30287 ts_tablet_manager.cc:1397] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:38:13.390245 30291 tablet_bootstrap.cc:492] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: No bootstrap required, opened a new log
I20260620 18:38:13.390365 30291 ts_tablet_manager.cc:1397] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:13.391295 30287 raft_consensus.cc:348] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.391419 30287 raft_consensus.cc:374] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:13.391440 30287 raft_consensus.cc:729] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c776262d53a4e129975c240c8afa581, State: Initialized, Role: FOLLOWER
I20260620 18:38:13.391559 30287 consensus_queue.cc:260] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.391816 30287 ts_tablet_manager.cc:1428] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:38:13.391866 30002 heartbeater.cc:503] Master 127.10.109.126:45619 was elected leader, sending a full tablet report...
I20260620 18:38:13.391912 30291 raft_consensus.cc:348] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.392068 30291 raft_consensus.cc:374] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:13.392127 30291 raft_consensus.cc:729] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a3dd65bc2843ba889ac6fc749550b5, State: Initialized, Role: FOLLOWER
I20260620 18:38:13.392238 30291 consensus_queue.cc:260] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.392534 30136 heartbeater.cc:503] Master 127.10.109.126:45619 was elected leader, sending a full tablet report...
I20260620 18:38:13.392740 30291 ts_tablet_manager.cc:1428] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 18:38:13.443681 30137 tablet.cc:2369] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:13.541494 30003 tablet.cc:2369] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:13.572659 30294 raft_consensus.cc:482] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:13.572774 30294 raft_consensus.cc:504] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.573118 30294 leader_election.cc:288] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4a3dd65bc2843ba889ac6fc749550b5 (127.10.109.66:39169), ccbf4a75d63944d09570642995cabb5d (127.10.109.67:36759)
I20260620 18:38:13.576059 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.573374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:34261 (local address 127.10.109.67:36759)
0620 18:38:13.573421 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.573422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.573505 (+ 83us) server_negotiation.cc:407] Connection header received
0620 18:38:13.573573 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.573574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.573589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.573672 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:13.573846 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.574298 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575300 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.575686 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575788 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.575797 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.575953 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.575990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:38:13.576067 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.573317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:45003 (local address 127.10.109.66:39169)
0620 18:38:13.573384 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.573386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.573441 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:38:13.573530 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.573530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.573538 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.573594 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:13.573773 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.574275 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575342 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.575706 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575796 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.575804 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.575961 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.575999 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:38:13.576081 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.573303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:36759 (local address 127.10.109.65:34261)
0620 18:38:13.573425 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.573435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.573515 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.573681 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.573681 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.573683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:13.573775 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.573777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.574375 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.574376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.575200 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.575203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.575795 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.575874 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.575877 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.575949 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.575990 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260620 18:38:13.576092 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.573245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:39169 (local address 127.10.109.65:45003)
0620 18:38:13.573379 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.573391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.573459 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.573598 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.573599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.573601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:13.573673 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.573675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.574357 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.574358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.575254 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.575256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.575798 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.575799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.575881 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.575884 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.575953 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.575996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260620 18:38:13.576532 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b3616b06c74e70893e77d49f2f6fd2" candidate_uuid: "1c776262d53a4e129975c240c8afa581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbf4a75d63944d09570642995cabb5d" is_pre_election: true
I20260620 18:38:13.576675 30224 raft_consensus.cc:2454] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c776262d53a4e129975c240c8afa581 in term 0.
I20260620 18:38:13.576660 30090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b3616b06c74e70893e77d49f2f6fd2" candidate_uuid: "1c776262d53a4e129975c240c8afa581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" is_pre_election: true
I20260620 18:38:13.576833 30090 raft_consensus.cc:2454] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c776262d53a4e129975c240c8afa581 in term 0.
I20260620 18:38:13.576929 29890 leader_election.cc:302] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c776262d53a4e129975c240c8afa581, ccbf4a75d63944d09570642995cabb5d; no voters:
I20260620 18:38:13.577049 30294 raft_consensus.cc:2781] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:13.577122 30294 raft_consensus.cc:482] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:13.577155 30294 raft_consensus.cc:3037] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:13.577884 30294 raft_consensus.cc:504] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.578033 30294 leader_election.cc:288] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [CANDIDATE]: Term 1 election: Requested vote from peers c4a3dd65bc2843ba889ac6fc749550b5 (127.10.109.66:39169), ccbf4a75d63944d09570642995cabb5d (127.10.109.67:36759)
I20260620 18:38:13.578176 30090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b3616b06c74e70893e77d49f2f6fd2" candidate_uuid: "1c776262d53a4e129975c240c8afa581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a3dd65bc2843ba889ac6fc749550b5"
I20260620 18:38:13.578188 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b3616b06c74e70893e77d49f2f6fd2" candidate_uuid: "1c776262d53a4e129975c240c8afa581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbf4a75d63944d09570642995cabb5d"
I20260620 18:38:13.578258 30090 raft_consensus.cc:3037] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:13.578271 30224 raft_consensus.cc:3037] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:13.579013 30224 raft_consensus.cc:2454] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c776262d53a4e129975c240c8afa581 in term 1.
I20260620 18:38:13.579064 30090 raft_consensus.cc:2454] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c776262d53a4e129975c240c8afa581 in term 1.
I20260620 18:38:13.579166 29890 leader_election.cc:302] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c776262d53a4e129975c240c8afa581, ccbf4a75d63944d09570642995cabb5d; no voters:
I20260620 18:38:13.579308 30294 raft_consensus.cc:2781] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:13.579428 30294 raft_consensus.cc:686] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 1 LEADER]: Becoming Leader. State: Replica: 1c776262d53a4e129975c240c8afa581, State: Running, Role: LEADER
I20260620 18:38:13.579515 30294 consensus_queue.cc:237] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } }
I20260620 18:38:13.580250 29810 catalog_manager.cc:5495] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c776262d53a4e129975c240c8afa581 (127.10.109.65). New cstate: current_term: 1 leader_uuid: "1c776262d53a4e129975c240c8afa581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c776262d53a4e129975c240c8afa581" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 35683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 } health_report { overall_health: UNKNOWN } } }
W20260620 18:38:13.598785 30271 tablet.cc:2369] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:13.605085 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.602977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.10.109.65:35683)
0620 18:38:13.603029 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.603031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.603039 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:13.603108 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.603109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.603124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.603181 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:13.603317 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.603769 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.604485 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.604664 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.604719 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.604844 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:13.604929 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:13.604986 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.605019 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.605035 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:38:13.606791 30090 raft_consensus.cc:1261] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 1 FOLLOWER]: Refusing update from remote peer 1c776262d53a4e129975c240c8afa581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:13.606803 30224 raft_consensus.cc:1261] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 1 FOLLOWER]: Refusing update from remote peer 1c776262d53a4e129975c240c8afa581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:13.607113 30294 consensus_queue.cc:1035] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a3dd65bc2843ba889ac6fc749550b5" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 39169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:13.607276 30298 consensus_queue.cc:1035] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbf4a75d63944d09570642995cabb5d" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:13.609547 30301 mvcc.cc:205] Tried to move back new op lower bound from 7298992921010909184 to 7298992920902225920. Current Snapshot: MvccSnapshot[applied={T|T < 7298992921010909184}]
I20260620 18:38:13.609793 30302 mvcc.cc:205] Tried to move back new op lower bound from 7298992921010909184 to 7298992920902225920. Current Snapshot: MvccSnapshot[applied={T|T < 7298992921010909184}]
I20260620 18:38:13.612330 29811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:13.612421 29811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:13.613147 29811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4797046f09924a3a81049500c06ed2e9: Sending DeleteTablet for 3 replicas of tablet 79b3616b06c74e70893e77d49f2f6fd2
I20260620 18:38:13.613425 29936 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:38052
I20260620 18:38:13.613449 30070 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:51744
I20260620 18:38:13.613626 30311 tablet_replica.cc:330] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581: stopping tablet replica
I20260620 18:38:13.613680 30204 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b3616b06c74e70893e77d49f2f6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.10.109.126@KRBTEST.COM'} at 127.0.0.1:41624
I20260620 18:38:13.613725 30311 raft_consensus.cc:2229] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:13.613806 30311 raft_consensus.cc:2258] T 79b3616b06c74e70893e77d49f2f6fd2 P 1c776262d53a4e129975c240c8afa581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:13.613924 30313 tablet_replica.cc:330] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: stopping tablet replica
I20260620 18:38:13.614037 30312 tablet_replica.cc:330] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: stopping tablet replica
I20260620 18:38:13.614063 30313 raft_consensus.cc:2229] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:13.614125 30312 raft_consensus.cc:2229] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:13.614182 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29873
I20260620 18:38:13.614218 30312 raft_consensus.cc:2258] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:13.614301 30313 raft_consensus.cc:2258] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:13.614907 30313 ts_tablet_manager.cc:1905] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:13.616257 30312 ts_tablet_manager.cc:1905] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:13.616569 30313 ts_tablet_manager.cc:1918] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:13.616645 30313 log.cc:1196] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/79b3616b06c74e70893e77d49f2f6fd2
I20260620 18:38:13.616933 30313 ts_tablet_manager.cc:1939] T 79b3616b06c74e70893e77d49f2f6fd2 P ccbf4a75d63944d09570642995cabb5d: Deleting consensus metadata
I20260620 18:38:13.617355 29796 catalog_manager.cc:4841] TS ccbf4a75d63944d09570642995cabb5d (127.10.109.67:36759): tablet 79b3616b06c74e70893e77d49f2f6fd2 (table test-table [id=318613b5e686403bb8f92cd32739fd5d]) successfully deleted
I20260620 18:38:13.618188 30312 ts_tablet_manager.cc:1918] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:13.618264 30312 log.cc:1196] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/79b3616b06c74e70893e77d49f2f6fd2
I20260620 18:38:13.618530 30312 ts_tablet_manager.cc:1939] T 79b3616b06c74e70893e77d49f2f6fd2 P c4a3dd65bc2843ba889ac6fc749550b5: Deleting consensus metadata
I20260620 18:38:13.619066 29795 catalog_manager.cc:4841] TS c4a3dd65bc2843ba889ac6fc749550b5 (127.10.109.66:39169): tablet 79b3616b06c74e70893e77d49f2f6fd2 (table test-table [id=318613b5e686403bb8f92cd32739fd5d]) successfully deleted
W20260620 18:38:13.620682 29796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:35683 (error 108)
I20260620 18:38:13.620873 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30007
I20260620 18:38:13.621735 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.621592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:35683 (local address 127.0.0.1:38068)
0620 18:38:13.621651 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.621694 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:38:13.621865 29796 catalog_manager.cc:4584] TS 1c776262d53a4e129975c240c8afa581 (127.10.109.65:35683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79b3616b06c74e70893e77d49f2f6fd2: Network error: Client connection negotiation failed: client connection to 127.10.109.65:35683: connect: Connection refused (error 111)
I20260620 18:38:13.625667 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30141
I20260620 18:38:13.629861 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 29779
2026-06-20T18:38:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2073 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:13Z Disabled control of system clock
I20260620 18:38:13.657001 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41133
--webserver_interface=127.10.109.126
--webserver_port=0
--builtin_ntp_servers=127.10.109.84:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:41133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:13.736063 30318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:13.736256 30318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:13.736279 30318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:13.737361 30318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:13.737396 30318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:13.737411 30318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:13.737422 30318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:13.737434 30318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:13.738679 30318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:41133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:13.738897 30318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:13.739130 30318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:13.743716 30326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.743752 30324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.743878 30323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:13.743970 30318 server_base.cc:1034] running on GCE node
I20260620 18:38:13.744233 30318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:13.744503 30318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:13.745666 30318 hybrid_clock.cc:648] HybridClock initialized: now 1781980693745636 us; error 48 us; skew 500 ppm
I20260620 18:38:13.746855 30318 webserver.cc:460] Webserver started at http://127.10.109.126:46317/ using document root <none> and password file <none>
I20260620 18:38:13.747136 30318 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:13.747207 30318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:13.747349 30318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:13.748389 30318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "1bc4e612ef6e4773940ace974de4decf"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "003b5a2416c15bba3dcae8b33a89b5c8"
server_key_iv: "3923679163e25fd8c7d53c072d8443e2"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.748720 30318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "1bc4e612ef6e4773940ace974de4decf"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "003b5a2416c15bba3dcae8b33a89b5c8"
server_key_iv: "3923679163e25fd8c7d53c072d8443e2"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.750263 30318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:13.751010 30332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.751212 30318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:13.751310 30318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "1bc4e612ef6e4773940ace974de4decf"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "003b5a2416c15bba3dcae8b33a89b5c8"
server_key_iv: "3923679163e25fd8c7d53c072d8443e2"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.751379 30318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:13.764561 30318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:13.764830 30318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:13.764966 30318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:13.768708 30318 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:41133
I20260620 18:38:13.768743 30384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:41133 every 8 connection(s)
I20260620 18:38:13.769047 30318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:13.769914 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:13.770812 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30318
I20260620 18:38:13.770926 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:13.771103 10677 external_mini_cluster.cc:1428] Setting key 2a11700e3ceb719017e0c29910a39fe2
I20260620 18:38:13.772460 30385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: Bootstrap starting.
I20260620 18:38:13.773077 30385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:13.773348 30385 log.cc:824] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:13.773969 30385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: No bootstrap required, opened a new log
I20260620 18:38:13.775873 30385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } }
I20260620 18:38:13.776021 30385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:13.776070 30385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc4e612ef6e4773940ace974de4decf, State: Initialized, Role: FOLLOWER
I20260620 18:38:13.776193 30385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } }
I20260620 18:38:13.776283 30385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:13.776326 30385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:13.776415 30385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:13.776571 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.771611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.10.109.126:41133)
0620 18:38:13.771793 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.771799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.771822 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:38:13.772288 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.772294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.772347 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.772457 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.772679 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.773624 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.774447 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.774891 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.775022 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.775773 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.775777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.775777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:13.775926 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.775955 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:13.775965 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.776035 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.776203 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.776292 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 18:38:13.777064 30385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } }
I20260620 18:38:13.777163 30385 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc4e612ef6e4773940ace974de4decf; no voters:
I20260620 18:38:13.777315 30385 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:13.777343 30390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:13.777472 30390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [term 1 LEADER]: Becoming Leader. State: Replica: 1bc4e612ef6e4773940ace974de4decf, State: Running, Role: LEADER
I20260620 18:38:13.777570 30385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:13.777549 30390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } }
I20260620 18:38:13.777974 30392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc4e612ef6e4773940ace974de4decf. Latest consensus state: current_term: 1 leader_uuid: "1bc4e612ef6e4773940ace974de4decf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } } }
I20260620 18:38:13.777982 30391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc4e612ef6e4773940ace974de4decf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc4e612ef6e4773940ace974de4decf" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41133 } } }
I20260620 18:38:13.778102 30392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:13.778111 30391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:13.778600 30399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:13.779065 30399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:13.780553 30399 catalog_manager.cc:1347] Generated new cluster ID: f5690e223b4248e6992a5141b093c215
I20260620 18:38:13.780602 30399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:13.794205 30399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:13.794687 30399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:13.809554 30399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: Generated new TSK 0
I20260620 18:38:13.809753 30399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:13.819026 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.65:0
--local_ip_for_outbound_sockets=127.10.109.65
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--builtin_ntp_servers=127.10.109.84:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:13.898051 30409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:13.898247 30409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:13.898269 30409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:13.899335 30409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:13.899366 30409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:13.899391 30409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.65
I20260620 18:38:13.900614 30409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.109.65
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.65
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:13.900789 30409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:13.901012 30409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:13.905748 30417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.905800 30414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:13.905915 30415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:13.905915 30409 server_base.cc:1034] running on GCE node
I20260620 18:38:13.906172 30409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:13.906387 30409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:13.907518 30409 hybrid_clock.cc:648] HybridClock initialized: now 1781980693907506 us; error 27 us; skew 500 ppm
I20260620 18:38:13.908656 30409 webserver.cc:460] Webserver started at http://127.10.109.65:39815/ using document root <none> and password file <none>
I20260620 18:38:13.908882 30409 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:13.908947 30409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:13.909094 30409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:13.910104 30409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data/instance:
uuid: "3dd0321ee44e4e7486764719453dc2f7"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "c2573b30c8e8b08d10f019074948aac5"
server_key_iv: "8aeafcef1a9e6ae2e2055a2d12e95489"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.910458 30409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance:
uuid: "3dd0321ee44e4e7486764719453dc2f7"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "c2573b30c8e8b08d10f019074948aac5"
server_key_iv: "8aeafcef1a9e6ae2e2055a2d12e95489"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.911808 30409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:13.912532 30423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.912711 30409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:13.912802 30409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal
uuid: "3dd0321ee44e4e7486764719453dc2f7"
format_stamp: "Formatted at 2026-06-20 18:38:13 on dist-test-slave-3jql"
server_key: "c2573b30c8e8b08d10f019074948aac5"
server_key_iv: "8aeafcef1a9e6ae2e2055a2d12e95489"
server_key_version: "encryptionkey@0"
I20260620 18:38:13.912882 30409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:13.925266 30409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:13.925526 30409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:13.925668 30409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:13.925875 30409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:13.926173 30409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:13.926229 30409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.926263 30409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:13.926290 30409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:13.931870 30409 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.65:40869
I20260620 18:38:13.931917 30536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.65:40869 every 8 connection(s)
I20260620 18:38:13.932212 30409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:13.933545 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30409
I20260620 18:38:13.933641 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:13.933835 10677 external_mini_cluster.cc:1428] Setting key e87d111ae2c29aa73ada332d636280ef
I20260620 18:38:13.934854 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.66:0
--local_ip_for_outbound_sockets=127.10.109.66
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--builtin_ntp_servers=127.10.109.84:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:38:13.937853 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.932905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.65:50909 (local address 127.10.109.126:41133)
0620 18:38:13.932952 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:13.932955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:13.933195 (+ 240us) server_negotiation.cc:407] Connection header received
0620 18:38:13.933691 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:13.933693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:13.933711 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:13.933764 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:13.934548 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.935321 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.936386 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.936614 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.936695 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.937524 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:13.937526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:13.937527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:13.937532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:13.937548 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:13.937553 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:13.937632 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:13.937742 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:38:13.937771 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:13.938086 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:13.932897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41133 (local address 127.10.109.65:50909)
0620 18:38:13.933081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:13.933112 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:13.933568 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:13.933841 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:13.933845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:13.933854 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:13.934450 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.934456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.935443 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.935444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.936258 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:13.936261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:13.936720 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:13.936722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:13.936846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:13.937379 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:13.937379 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:13.937405 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:13.937426 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:13.937636 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:13.937642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:13.937645 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:13.937742 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:38:13.937841 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260620 18:38:13.938552 30537 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41133
I20260620 18:38:13.938642 30537 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:13.938863 30537 heartbeater.cc:511] Master 127.10.109.126:41133 requested a full tablet report, sending...
I20260620 18:38:13.939359 30349 ts_manager.cc:194] Registered new tserver with Master: 3dd0321ee44e4e7486764719453dc2f7 (127.10.109.65:40869)
I20260620 18:38:13.939975 30349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.65:50909
W20260620 18:38:14.014025 30540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:14.014245 30540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:14.014279 30540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:14.015516 30540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:14.015563 30540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:14.015595 30540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.66
I20260620 18:38:14.016973 30540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.109.66
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.66
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:14.017140 30540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:14.017356 30540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:14.022131 30545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:14.022249 30546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:14.022293 30548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:14.022333 30540 server_base.cc:1034] running on GCE node
I20260620 18:38:14.022535 30540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:14.022727 30540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:14.023923 30540 hybrid_clock.cc:648] HybridClock initialized: now 1781980694023903 us; error 34 us; skew 500 ppm
I20260620 18:38:14.025171 30540 webserver.cc:460] Webserver started at http://127.10.109.66:40721/ using document root <none> and password file <none>
I20260620 18:38:14.025414 30540 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:14.025462 30540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:14.025595 30540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:14.026791 30540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data/instance:
uuid: "27d773e539474236b962c6bc648cc1b8"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "806ff4ccad23c99495e466fde101dcc6"
server_key_iv: "ebfe0d2ebb506380c3026cf4200044b0"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.027189 30540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance:
uuid: "27d773e539474236b962c6bc648cc1b8"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "806ff4ccad23c99495e466fde101dcc6"
server_key_iv: "ebfe0d2ebb506380c3026cf4200044b0"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.028425 30540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:14.029145 30554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.029311 30540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:14.029394 30540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal
uuid: "27d773e539474236b962c6bc648cc1b8"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "806ff4ccad23c99495e466fde101dcc6"
server_key_iv: "ebfe0d2ebb506380c3026cf4200044b0"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.029446 30540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:14.036664 30540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:14.036875 30540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:14.036968 30540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:14.037128 30540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:14.037434 30540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:14.037469 30540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.037490 30540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:14.037533 30540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.043035 30540 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.66:36943
I20260620 18:38:14.043100 30667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.66:36943 every 8 connection(s)
I20260620 18:38:14.043393 30540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:14.048120 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.044058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:41111 (local address 127.10.109.126:41133)
0620 18:38:14.044131 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.044133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.044326 (+ 193us) server_negotiation.cc:407] Connection header received
0620 18:38:14.044771 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.044772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.044782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.044828 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:14.045420 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.045915 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.046903 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.047090 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.047155 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.047840 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:14.047841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:14.047842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:14.047846 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:14.047859 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:14.047863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:14.047911 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.048040 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.048062 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:14.048197 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.044022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41133 (local address 127.10.109.66:41111)
0620 18:38:14.044192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.044250 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.044673 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.044883 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.044886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.044893 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:14.045357 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.045360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.045991 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.045992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.046829 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.046831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.047165 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.047165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.047287 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.047724 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:14.047724 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:14.047748 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:14.047768 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:14.047914 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:14.047916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:14.047918 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.047961 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.048027 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260620 18:38:14.048700 30668 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41133
I20260620 18:38:14.048782 30668 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:14.048913 30668 heartbeater.cc:511] Master 127.10.109.126:41133 requested a full tablet report, sending...
I20260620 18:38:14.049254 30349 ts_manager.cc:194] Registered new tserver with Master: 27d773e539474236b962c6bc648cc1b8 (127.10.109.66:36943)
I20260620 18:38:14.049412 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30540
I20260620 18:38:14.049516 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:14.049666 30349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.66:41111
I20260620 18:38:14.049717 10677 external_mini_cluster.cc:1428] Setting key aa45dee68709e3bebfce4cd7cb2bf6ec
I20260620 18:38:14.050629 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.109.67:0
--local_ip_for_outbound_sockets=127.10.109.67
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--builtin_ntp_servers=127.10.109.84:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:38:14.128473 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:14.128675 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:14.128707 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:14.130247 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:14.130295 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:14.130336 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.109.67
I20260620 18:38:14.131680 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.109.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.109.67
--webserver_port=0
--tserver_master_addrs=127.10.109.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.109.67
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:14.131965 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:14.132251 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:14.137115 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:14.137138 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:14.137120 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:14.137321 30671 server_base.cc:1034] running on GCE node
I20260620 18:38:14.137563 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:14.137754 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:14.138886 30671 hybrid_clock.cc:648] HybridClock initialized: now 1781980694138853 us; error 48 us; skew 500 ppm
I20260620 18:38:14.140096 30671 webserver.cc:460] Webserver started at http://127.10.109.67:38091/ using document root <none> and password file <none>
I20260620 18:38:14.140328 30671 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:14.140372 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:14.140501 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:14.141594 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data/instance:
uuid: "8a1398e3bf184e7f95d94b7a615421c7"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "654995fb916432a7f25ebe6e2d4903da"
server_key_iv: "e905f67df6cf02001f1a8e825e1395bd"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.141932 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance:
uuid: "8a1398e3bf184e7f95d94b7a615421c7"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "654995fb916432a7f25ebe6e2d4903da"
server_key_iv: "e905f67df6cf02001f1a8e825e1395bd"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.143255 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:14.143939 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.144106 30671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:14.144202 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal
uuid: "8a1398e3bf184e7f95d94b7a615421c7"
format_stamp: "Formatted at 2026-06-20 18:38:14 on dist-test-slave-3jql"
server_key: "654995fb916432a7f25ebe6e2d4903da"
server_key_iv: "e905f67df6cf02001f1a8e825e1395bd"
server_key_version: "encryptionkey@0"
I20260620 18:38:14.144272 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:14.155259 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:14.155511 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:14.155644 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:14.155846 30671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:14.156137 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:14.156199 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.156234 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:14.156260 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:14.161938 30671 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.67:42883
I20260620 18:38:14.161972 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.67:42883 every 8 connection(s)
I20260620 18:38:14.162286 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:14.165414 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30671
I20260620 18:38:14.165660 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:14.165973 10677 external_mini_cluster.cc:1428] Setting key 4f63bfd1bb4e188dd8749444076329f0
I20260620 18:38:14.168045 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.163003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.67:42759 (local address 127.10.109.126:41133)
0620 18:38:14.163059 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.163060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.163256 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:38:14.163696 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.163697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.163707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.163778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:14.164390 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.164903 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.166327 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.166554 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.166647 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.167722 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:14.167723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:14.167724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:14.167729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:14.167749 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:14.167754 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:14.167833 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.167935 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.167968 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:38:14.168264 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.162997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:41133 (local address 127.10.109.67:42759)
0620 18:38:14.163155 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.163188 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.163617 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.163827 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.163830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.163838 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:14.164299 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.164303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.164996 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.164997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.166132 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.166139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.166663 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.166665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.166922 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.167528 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:14.167529 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:14.167558 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:14.167584 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:14.167838 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:14.167843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:14.167847 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.167925 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.168036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260620 18:38:14.168679 30799 heartbeater.cc:347] Connected to a master server at 127.10.109.126:41133
I20260620 18:38:14.168759 30799 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:14.168892 30799 heartbeater.cc:511] Master 127.10.109.126:41133 requested a full tablet report, sending...
I20260620 18:38:14.169242 30349 ts_manager.cc:194] Registered new tserver with Master: 8a1398e3bf184e7f95d94b7a615421c7 (127.10.109.67:42883)
I20260620 18:38:14.169603 30349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.109.67:42759
I20260620 18:38:14.177713 10677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:14.181695 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.179164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.10.109.126:41133)
0620 18:38:14.179209 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.179211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.179257 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:14.179307 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.179308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.179315 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.179367 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:14.179478 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.179932 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.180642 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.180820 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.180873 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.181508 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:14.181509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:14.181509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:14.181513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:14.181526 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:14.181530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:14.181588 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.181625 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.181642 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:14.183287 30349 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:14.184851 30349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:14.191494 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.10.109.67:42883)
0620 18:38:14.187840 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.187844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.188068 (+ 224us) server_negotiation.cc:407] Connection header received
0620 18:38:14.188186 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.188188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.188203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.188258 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:14.188494 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.189189 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.190372 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.191019 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.191156 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.191248 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.191396 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.191431 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260620 18:38:14.191628 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42883 (local address 127.0.0.1:47606)
0620 18:38:14.187993 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.188004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.188093 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.188345 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.188347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.188349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:14.188416 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.188418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.189275 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.189276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.190307 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.190310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.191164 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.191165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.191248 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.191262 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.191365 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.191500 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":75,"thread_start_us":87,"threads_started":1}
I20260620 18:38:14.192312 30733 tablet_service.cc:1467] Processing CreateTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b (DEFAULT_TABLE table=test-table [id=b9bfac28f9914e848292c52d245e17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.192684 30733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566e1ab6badd4516b29ffc1481f6c93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.193463 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.66:36943 (local address 127.0.0.1:40862)
0620 18:38:14.187647 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.187699 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.187769 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.188350 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.188351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.188353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:14.188417 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.188419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.191304 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.191304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.192413 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.192416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.193242 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.193243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.193305 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.193309 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.193360 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.193398 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":52,"threads_started":1}
I20260620 18:38:14.193497 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.10.109.66:36943)
0620 18:38:14.188045 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.188049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.188129 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:38:14.188173 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.188175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.188202 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.188278 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:14.190075 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.191231 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.192492 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.193149 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.193239 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.193325 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.193383 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.193418 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260620 18:38:14.193845 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40869 (local address 127.0.0.1:42142)
0620 18:38:14.188012 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.188023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.188070 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.188344 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.188347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.188352 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:14.188417 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.188418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.190057 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.190058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.191231 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.191234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.193610 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.193611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.193673 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.193676 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.193737 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.193777 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260620 18:38:14.193995 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.187802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.10.109.65:40869)
0620 18:38:14.187962 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.187965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.188144 (+ 179us) server_negotiation.cc:407] Connection header received
0620 18:38:14.188187 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.188189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.188203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.188273 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:14.189278 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.189965 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.192523 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.193371 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.193444 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.193521 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.193864 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.193915 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260620 18:38:14.194393 30602 tablet_service.cc:1467] Processing CreateTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b (DEFAULT_TABLE table=test-table [id=b9bfac28f9914e848292c52d245e17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.194574 30471 tablet_service.cc:1467] Processing CreateTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b (DEFAULT_TABLE table=test-table [id=b9bfac28f9914e848292c52d245e17e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.194726 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566e1ab6badd4516b29ffc1481f6c93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.194936 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566e1ab6badd4516b29ffc1481f6c93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.196089 30813 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Bootstrap starting.
I20260620 18:38:14.196815 30813 tablet_bootstrap.cc:654] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.197156 30813 log.cc:824] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:14.197707 30815 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Bootstrap starting.
I20260620 18:38:14.197933 30813 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: No bootstrap required, opened a new log
I20260620 18:38:14.198004 30813 ts_tablet_manager.cc:1397] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:14.198114 30816 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: Bootstrap starting.
I20260620 18:38:14.198482 30815 tablet_bootstrap.cc:654] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.198722 30816 tablet_bootstrap.cc:654] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.198835 30815 log.cc:824] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:14.199088 30816 log.cc:824] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:14.199605 30815 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: No bootstrap required, opened a new log
I20260620 18:38:14.199683 30815 ts_tablet_manager.cc:1397] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:14.199810 30816 tablet_bootstrap.cc:492] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: No bootstrap required, opened a new log
I20260620 18:38:14.199880 30816 ts_tablet_manager.cc:1397] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:14.200884 30813 raft_consensus.cc:348] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.201049 30813 raft_consensus.cc:374] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.201085 30813 raft_consensus.cc:729] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1398e3bf184e7f95d94b7a615421c7, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.201233 30813 consensus_queue.cc:260] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.201485 30813 ts_tablet_manager.cc:1428] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:14.201521 30799 heartbeater.cc:503] Master 127.10.109.126:41133 was elected leader, sending a full tablet report...
I20260620 18:38:14.201683 30815 raft_consensus.cc:348] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.201831 30815 raft_consensus.cc:374] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.201872 30815 raft_consensus.cc:729] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d773e539474236b962c6bc648cc1b8, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.201983 30815 consensus_queue.cc:260] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.202230 30668 heartbeater.cc:503] Master 127.10.109.126:41133 was elected leader, sending a full tablet report...
I20260620 18:38:14.202345 30816 raft_consensus.cc:348] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.202216 30815 ts_tablet_manager.cc:1428] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:38:14.202513 30816 raft_consensus.cc:374] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.202564 30816 raft_consensus.cc:729] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd0321ee44e4e7486764719453dc2f7, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.202674 30816 consensus_queue.cc:260] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.202951 30816 ts_tablet_manager.cc:1428] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:14.203092 30537 heartbeater.cc:503] Master 127.10.109.126:41133 was elected leader, sending a full tablet report...
W20260620 18:38:14.293919 30669 tablet.cc:2369] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:14.324604 30820 raft_consensus.cc:482] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:14.324738 30820 raft_consensus.cc:504] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.325125 30820 leader_election.cc:288] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a1398e3bf184e7f95d94b7a615421c7 (127.10.109.67:42883), 3dd0321ee44e4e7486764719453dc2f7 (127.10.109.65:40869)
I20260620 18:38:14.328944 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.325278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.67:42883 (local address 127.10.109.66:56463)
0620 18:38:14.325348 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.325363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.325422 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.325588 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.325589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.325591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:14.325663 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.325665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.326607 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.326607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.327892 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.327895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.328634 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.328635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.328725 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.328731 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.328803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.328846 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":33}
I20260620 18:38:14.328953 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.325354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:56463 (local address 127.10.109.67:42883)
0620 18:38:14.325445 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.325447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.325457 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:14.325493 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.325494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.325504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.325557 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:14.325766 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.326529 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.327984 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.328522 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.328619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.328627 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.328840 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.328885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:38:14.328953 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.325464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.109.66:46323 (local address 127.10.109.65:40869)
0620 18:38:14.325521 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.325523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.325640 (+ 117us) server_negotiation.cc:407] Connection header received
0620 18:38:14.325664 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.325665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.325675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.325741 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:14.325892 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.326620 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.327985 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.328531 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.328602 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.328611 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.328842 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.328887 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:38:14.328951 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.325398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40869 (local address 127.10.109.66:46323)
0620 18:38:14.325550 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.325560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:14.325615 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:14.325757 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:14.325758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:14.325760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:14.325822 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.325823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.326687 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.326687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.327902 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:14.327904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.328634 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.328635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:14.328725 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.328731 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:38:14.328803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:14.328846 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":18,"thread_start_us":88,"threads_started":1}
I20260620 18:38:14.329319 30491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566e1ab6badd4516b29ffc1481f6c93b" candidate_uuid: "27d773e539474236b962c6bc648cc1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd0321ee44e4e7486764719453dc2f7" is_pre_election: true
I20260620 18:38:14.329319 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566e1ab6badd4516b29ffc1481f6c93b" candidate_uuid: "27d773e539474236b962c6bc648cc1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1398e3bf184e7f95d94b7a615421c7" is_pre_election: true
I20260620 18:38:14.329465 30753 raft_consensus.cc:2454] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d773e539474236b962c6bc648cc1b8 in term 0.
I20260620 18:38:14.329465 30491 raft_consensus.cc:2454] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d773e539474236b962c6bc648cc1b8 in term 0.
I20260620 18:38:14.329705 30556 leader_election.cc:302] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d773e539474236b962c6bc648cc1b8, 8a1398e3bf184e7f95d94b7a615421c7; no voters:
I20260620 18:38:14.329842 30820 raft_consensus.cc:2781] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:14.329898 30820 raft_consensus.cc:482] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:14.329923 30820 raft_consensus.cc:3037] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.330617 30820 raft_consensus.cc:504] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.330739 30820 leader_election.cc:288] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a1398e3bf184e7f95d94b7a615421c7 (127.10.109.67:42883), 3dd0321ee44e4e7486764719453dc2f7 (127.10.109.65:40869)
I20260620 18:38:14.330971 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566e1ab6badd4516b29ffc1481f6c93b" candidate_uuid: "27d773e539474236b962c6bc648cc1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1398e3bf184e7f95d94b7a615421c7"
I20260620 18:38:14.330969 30491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566e1ab6badd4516b29ffc1481f6c93b" candidate_uuid: "27d773e539474236b962c6bc648cc1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd0321ee44e4e7486764719453dc2f7"
I20260620 18:38:14.331039 30753 raft_consensus.cc:3037] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.331060 30491 raft_consensus.cc:3037] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.331815 30753 raft_consensus.cc:2454] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d773e539474236b962c6bc648cc1b8 in term 1.
I20260620 18:38:14.331830 30491 raft_consensus.cc:2454] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d773e539474236b962c6bc648cc1b8 in term 1.
I20260620 18:38:14.331972 30556 leader_election.cc:302] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d773e539474236b962c6bc648cc1b8, 3dd0321ee44e4e7486764719453dc2f7; no voters:
I20260620 18:38:14.332126 30820 raft_consensus.cc:2781] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:14.332271 30820 raft_consensus.cc:686] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 1 LEADER]: Becoming Leader. State: Replica: 27d773e539474236b962c6bc648cc1b8, State: Running, Role: LEADER
I20260620 18:38:14.332365 30820 consensus_queue.cc:237] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } }
I20260620 18:38:14.333129 30348 catalog_manager.cc:5495] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27d773e539474236b962c6bc648cc1b8 (127.10.109.66). New cstate: current_term: 1 leader_uuid: "27d773e539474236b962c6bc648cc1b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d773e539474236b962c6bc648cc1b8" member_type: VOTER last_known_addr { host: "127.10.109.66" port: 36943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:14.372365 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.370234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.10.109.66:36943)
0620 18:38:14.370313 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.370315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.370329 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:38:14.370361 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.370362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.370378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.370444 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:14.370560 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.371030 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.371746 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.371921 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.371970 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.372098 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:14.372185 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:14.372243 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.372276 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.372290 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:38:14.374064 30753 raft_consensus.cc:1261] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 1 FOLLOWER]: Refusing update from remote peer 27d773e539474236b962c6bc648cc1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.374060 30491 raft_consensus.cc:1261] T 566e1ab6badd4516b29ffc1481f6c93b P 3dd0321ee44e4e7486764719453dc2f7 [term 1 FOLLOWER]: Refusing update from remote peer 27d773e539474236b962c6bc648cc1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.374441 30820 consensus_queue.cc:1035] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1398e3bf184e7f95d94b7a615421c7" member_type: VOTER last_known_addr { host: "127.10.109.67" port: 42883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:14.374585 30823 consensus_queue.cc:1035] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0321ee44e4e7486764719453dc2f7" member_type: VOTER last_known_addr { host: "127.10.109.65" port: 40869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:14.376648 30826 mvcc.cc:205] Tried to move back new op lower bound from 7298992924153573376 to 7298992923986059264. Current Snapshot: MvccSnapshot[applied={T|T < 7298992924153573376}]
I20260620 18:38:14.379988 30348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:14.380071 30348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:14.380779 30348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bc4e612ef6e4773940ace974de4decf: Sending DeleteTablet for 3 replicas of tablet 566e1ab6badd4516b29ffc1481f6c93b
I20260620 18:38:14.381142 30602 tablet_service.cc:1514] Processing DeleteTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='slave'} at 127.0.0.1:40862
I20260620 18:38:14.381150 30733 tablet_service.cc:1514] Processing DeleteTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='slave'} at 127.0.0.1:47606
I20260620 18:38:14.381310 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30409
I20260620 18:38:14.381357 30471 tablet_service.cc:1514] Processing DeleteTablet for tablet 566e1ab6badd4516b29ffc1481f6c93b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='slave'} at 127.0.0.1:42142
I20260620 18:38:14.382055 30835 tablet_replica.cc:330] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: stopping tablet replica
I20260620 18:38:14.382190 30835 raft_consensus.cc:2229] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:14.382340 30835 raft_consensus.cc:2258] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.382527 30836 tablet_replica.cc:330] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: stopping tablet replica
I20260620 18:38:14.382687 30836 raft_consensus.cc:2229] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:14.382997 30835 ts_tablet_manager.cc:1905] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.384646 30836 raft_consensus.cc:2258] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.384917 30835 ts_tablet_manager.cc:1918] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:14.384984 30835 log.cc:1196] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-1/wal/wals/566e1ab6badd4516b29ffc1481f6c93b
I20260620 18:38:14.385258 30835 ts_tablet_manager.cc:1939] T 566e1ab6badd4516b29ffc1481f6c93b P 27d773e539474236b962c6bc648cc1b8: Deleting consensus metadata
I20260620 18:38:14.385645 30333 catalog_manager.cc:4841] TS 27d773e539474236b962c6bc648cc1b8 (127.10.109.66:36943): tablet 566e1ab6badd4516b29ffc1481f6c93b (table test-table [id=b9bfac28f9914e848292c52d245e17e8]) successfully deleted
I20260620 18:38:14.385751 30836 ts_tablet_manager.cc:1905] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:38:14.387100 30334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.109.65:40869 (error 108)
I20260620 18:38:14.387353 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30540
I20260620 18:38:14.387568 30836 ts_tablet_manager.cc:1918] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:14.387634 30836 log.cc:1196] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Deleting WAL directory at /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980452801806-10677-0/minicluster-data/ts-2/wal/wals/566e1ab6badd4516b29ffc1481f6c93b
I20260620 18:38:14.387866 30836 ts_tablet_manager.cc:1939] T 566e1ab6badd4516b29ffc1481f6c93b P 8a1398e3bf184e7f95d94b7a615421c7: Deleting consensus metadata
I20260620 18:38:14.388296 30334 catalog_manager.cc:4841] TS 8a1398e3bf184e7f95d94b7a615421c7 (127.10.109.67:42883): tablet 566e1ab6badd4516b29ffc1481f6c93b (table test-table [id=b9bfac28f9914e848292c52d245e17e8]) successfully deleted
I20260620 18:38:14.388345 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.388255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.65:40869 (local address 127.0.0.1:42158)
0620 18:38:14.388288 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:14.388320 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 18:38:14.388459 30334 catalog_manager.cc:4584] TS 3dd0321ee44e4e7486764719453dc2f7 (127.10.109.65:40869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 566e1ab6badd4516b29ffc1481f6c93b: Network error: Client connection negotiation failed: client connection to 127.10.109.65:40869: connect: Connection refused (error 111)
I20260620 18:38:14.393194 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30671
I20260620 18:38:14.397395 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30318
2026-06-20T18:38:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (766 ms)
[----------] 6 tests from EncryptionPolicyTest (8262 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:14 dist-test-slave-3jql krb5kdc[30840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:14 dist-test-slave-3jql krb5kdc[30840](info): set up 2 sockets
Jun 20 18:38:14 dist-test-slave-3jql krb5kdc[30840](info): commencing operation
krb5kdc: starting...
W20260620 18:38:15.438848 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:15.553625 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:15.631917 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:15.632097 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:15.632118 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:15.633221 30853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:15.633260 30853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:15.633273 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:15.633285 30853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:15.633297 30853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:15.634733 30853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:15.634999 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:15.635246 30853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:15.639809 30859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:15.639810 30861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:15.639822 30858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:15.639848 30853 server_base.cc:1034] running on GCE node
I20260620 18:38:15.640236 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:15.640491 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:15.641631 30853 hybrid_clock.cc:648] HybridClock initialized: now 1781980695641620 us; error 27 us; skew 500 ppm
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:15.643926 30853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:15.644361 30853 webserver.cc:460] Webserver started at http://127.0.0.1:34297/ using document root <none> and password file <none>
I20260620 18:38:15.644647 30853 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:15.644719 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:15.644858 30853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:15.645815 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "5aa008c323944095b7df430d3e328339"
format_stamp: "Formatted at 2026-06-20 18:38:15 on dist-test-slave-3jql"
server_key: "cf978a7a8b522866d60e330d393d8c3d"
server_key_iv: "ab4c635474db8f19e81b694cb19d8d23"
server_key_version: "encryptionkey@0"
I20260620 18:38:15.646143 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "5aa008c323944095b7df430d3e328339"
format_stamp: "Formatted at 2026-06-20 18:38:15 on dist-test-slave-3jql"
server_key: "cf978a7a8b522866d60e330d393d8c3d"
server_key_iv: "ab4c635474db8f19e81b694cb19d8d23"
server_key_version: "encryptionkey@0"
I20260620 18:38:15.647454 30853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 18:38:15.648150 30868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:15.648307 30853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:15.648397 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "5aa008c323944095b7df430d3e328339"
format_stamp: "Formatted at 2026-06-20 18:38:15 on dist-test-slave-3jql"
server_key: "cf978a7a8b522866d60e330d393d8c3d"
server_key_iv: "ab4c635474db8f19e81b694cb19d8d23"
server_key_version: "encryptionkey@0"
I20260620 18:38:15.648470 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:15.660044 30853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:15.661664 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:15.661824 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:15.665684 30853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36823
I20260620 18:38:15.665717 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36823 every 8 connection(s)
I20260620 18:38:15.666030 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:15.666903 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:15.668119 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30853
I20260620 18:38:15.668234 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:15.668401 10677 external_mini_cluster.cc:1428] Setting key e5bda050a178024cfc2419271317a617
I20260620 18:38:15.669443 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339: Bootstrap starting.
I20260620 18:38:15.670032 30921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:15.670325 30921 log.cc:824] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:15.671007 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339: No bootstrap required, opened a new log
I20260620 18:38:15.672906 30921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260620 18:38:15.673058 30921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:15.673100 30921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa008c323944095b7df430d3e328339, State: Initialized, Role: FOLLOWER
I20260620 18:38:15.673228 30921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260620 18:38:15.673316 30921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:15.673358 30921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:15.673404 30921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:15.673933 30921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260620 18:38:15.674046 30921 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa008c323944095b7df430d3e328339; no voters:
I20260620 18:38:15.674202 30921 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:15.674360 30926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:15.674460 30921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:15.674510 30926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa008c323944095b7df430d3e328339, State: Running, Role: LEADER
I20260620 18:38:15.674597 30926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260620 18:38:15.674925 30927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa008c323944095b7df430d3e328339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260620 18:38:15.675024 30927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:15.675189 30933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:15.675282 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa008c323944095b7df430d3e328339. Latest consensus state: current_term: 1 leader_uuid: "5aa008c323944095b7df430d3e328339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa008c323944095b7df430d3e328339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260620 18:38:15.675396 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:15.675678 30933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:38:15.679257 30933 catalog_manager.cc:1347] Generated new cluster ID: f520ea707c264b4cb4aafe362de65b03
I20260620 18:38:15.679306 30933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:15.680219 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:15.669211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51741 (local address 127.0.0.1:36823)
0620 18:38:15.669441 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:15.669452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:15.669489 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:38:15.670173 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:15.670184 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:15.670323 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:15.670518 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:15.671256 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:15.672109 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:15.673100 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:15.673659 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:15.673794 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:15.676808 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:15.676812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:15.676812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:15.676824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:15.678441 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:15.678954 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:15.678957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:15.678958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:15.679066 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:15.679395 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:15.679396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:15.679396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:15.679612 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:15.679770 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:15.679905 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:38:15.680002 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260620 18:38:15.681190 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:15.689769 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:15.682686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50041 (local address 127.0.0.1:36823)
0620 18:38:15.682722 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:15.682724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:15.682874 (+ 150us) server_negotiation.cc:407] Connection header received
0620 18:38:15.682917 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:15.682918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:15.682928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:15.682998 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:15.683560 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:15.684341 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:15.685305 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:15.685639 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:15.685708 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:15.687007 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:15.687008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:15.687009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:15.687015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:15.688643 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:15.689004 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:15.689006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:15.689006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:15.689068 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:15.689279 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:15.689279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:15.689280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:15.689404 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:15.689497 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:15.689615 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:38:15.689676 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:15.710244 30933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:15.710837 30933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:15.723666 30933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa008c323944095b7df430d3e328339: Generated new TSK 0
I20260620 18:38:15.723841 30933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:15.736219 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30853
W20260620 18:38:15.741094 30838 connection.cc:535] client connection to 127.0.0.1:36823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36823): Connection reset by peer (error 104)
2026-06-20T18:38:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30952](info): set up 2 sockets
Jun 20 18:38:15 dist-test-slave-3jql krb5kdc[30952](info): commencing operation
krb5kdc: starting...
W20260620 18:38:16.779063 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:16 dist-test-slave-3jql krb5kdc[30952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:16.914911 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:17.006490 30965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:17.006788 30965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:17.006832 30965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:17.008195 30965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:17.008242 30965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:17.008256 30965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:17.008270 30965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:17.008282 30965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:17.009778 30965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:17.010026 30965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:17.010222 30965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:17.015431 30970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:17.015527 30971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:17.015677 30973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:17.015901 30965 server_base.cc:1034] running on GCE node
I20260620 18:38:17.016275 30965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:17.016647 30965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:17.017848 30965 hybrid_clock.cc:648] HybridClock initialized: now 1781980697017827 us; error 42 us; skew 500 ppm
Jun 20 18:38:17 dist-test-slave-3jql krb5kdc[30952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:17.027137 30965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:17.027643 30965 webserver.cc:460] Webserver started at http://127.0.0.1:40151/ using document root <none> and password file <none>
I20260620 18:38:17.027910 30965 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:17.027985 30965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:17.028139 30965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:17.029299 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "8fa7df66ed3f4ccba4da9c77194a3a84"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-3jql"
server_key: "bfce905ad6f7ce7af0600864d64051f9"
server_key_iv: "4de014bdf16e00c5f60da2358e3dfc65"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.029850 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "8fa7df66ed3f4ccba4da9c77194a3a84"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-3jql"
server_key: "bfce905ad6f7ce7af0600864d64051f9"
server_key_iv: "4de014bdf16e00c5f60da2358e3dfc65"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.031361 30965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 18:38:17.032399 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:17.032691 30965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:17.032802 30965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "8fa7df66ed3f4ccba4da9c77194a3a84"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-3jql"
server_key: "bfce905ad6f7ce7af0600864d64051f9"
server_key_iv: "4de014bdf16e00c5f60da2358e3dfc65"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.032886 30965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:17.056847 30965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:17.059931 30965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:17.060061 30965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:17.064352 30965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40775
I20260620 18:38:17.064385 31032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40775 every 8 connection(s)
I20260620 18:38:17.064760 30965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:17.065907 31033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:17.068423 31033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84: Bootstrap starting.
I20260620 18:38:17.069038 31033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:17.069327 31033 log.cc:824] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:17.070096 31033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84: No bootstrap required, opened a new log
I20260620 18:38:17.070528 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 30965
I20260620 18:38:17.070637 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:17.070855 10677 external_mini_cluster.cc:1428] Setting key 95e4ba70fcdde450da4a224efc6a7bd3
I20260620 18:38:17.072443 31033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260620 18:38:17.072619 31033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:17.072657 31033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa7df66ed3f4ccba4da9c77194a3a84, State: Initialized, Role: FOLLOWER
I20260620 18:38:17.072789 31033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260620 18:38:17.072897 31033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:17.072929 31033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:17.072978 31033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:17.073736 31033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260620 18:38:17.073846 31033 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fa7df66ed3f4ccba4da9c77194a3a84; no voters:
I20260620 18:38:17.073997 31033 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:17.074055 31038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:17.074199 31038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa7df66ed3f4ccba4da9c77194a3a84, State: Running, Role: LEADER
I20260620 18:38:17.074239 31033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:17.074286 31038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } }
I20260620 18:38:17.074617 31038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa7df66ed3f4ccba4da9c77194a3a84. Latest consensus state: current_term: 1 leader_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } } }
I20260620 18:38:17.074640 31039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7df66ed3f4ccba4da9c77194a3a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40775 } } }
I20260620 18:38:17.074779 31039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:17.074708 31038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:17.075011 31046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:17.075526 31046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:17.077809 31046 catalog_manager.cc:1347] Generated new cluster ID: c3e88f152b844733b7a6411c955ca042
I20260620 18:38:17.077852 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:17.071406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57529 (local address 127.0.0.1:40775)
0620 18:38:17.071601 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:17.071611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:17.071644 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:38:17.072402 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:17.072411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:17.072477 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:17.072641 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:17.072880 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.074636 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.075937 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.076351 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.076465 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:17.077205 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:17.077209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:17.077209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:17.077388 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:17.077418 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:17.077428 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:17.077519 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:17.077600 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:38:17.077629 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260620 18:38:17.077867 31046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:17.078871 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:17.083649 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:17.080342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40261 (local address 127.0.0.1:40775)
0620 18:38:17.080390 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:17.080392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:17.080401 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:17.080453 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:17.080454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:17.080463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:17.080527 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:17.080651 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.081357 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.082308 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.082665 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.082759 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:17.083402 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:17.083403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:17.083403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:17.083407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:17.083424 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:17.083428 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:17.083487 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:17.083558 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:38:17.083570 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:38:17.085469 31046 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:17.086145 31046 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:17.096422 31046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fa7df66ed3f4ccba4da9c77194a3a84: Generated new TSK 0
I20260620 18:38:17.096609 31046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:17.103077 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 30965
W20260620 18:38:17.107709 30950 connection.cc:535] client connection to 127.0.0.1:40775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40775): Connection reset by peer (error 104)
2026-06-20T18:38:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:17 dist-test-slave-3jql krb5kdc[31064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:17 dist-test-slave-3jql krb5kdc[31064](info): set up 2 sockets
Jun 20 18:38:17 dist-test-slave-3jql krb5kdc[31064](info): commencing operation
krb5kdc: starting...
W20260620 18:38:18.175705 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:18 dist-test-slave-3jql krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:18.315327 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:18.400511 31077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:18.400722 31077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:18.400748 31077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:18.401897 31077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:18.401942 31077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:18.401957 31077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:18.401969 31077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:18.401985 31077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:18.403508 31077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:18.403750 31077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:18.403957 31077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:18.409461 31085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:18.409457 31082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:18.409531 31077 server_base.cc:1034] running on GCE node
W20260620 18:38:18.409457 31083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:18.409881 31077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:18.410149 31077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:18.411332 31077 hybrid_clock.cc:648] HybridClock initialized: now 1781980698411281 us; error 66 us; skew 500 ppm
Jun 20 18:38:18 dist-test-slave-3jql krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:18.413746 31077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:18.414148 31077 webserver.cc:460] Webserver started at http://127.0.0.1:33237/ using document root <none> and password file <none>
I20260620 18:38:18.414345 31077 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:18.414386 31077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:18.414470 31077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:18.415522 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "27fbb243b0324a879895634c6ffa2114"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-3jql"
server_key: "de3ed76247bf96c679f2be35b3b2ffce"
server_key_iv: "4b88585baf5d7ca8a2cd878369dc8aa6"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.415822 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "27fbb243b0324a879895634c6ffa2114"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-3jql"
server_key: "de3ed76247bf96c679f2be35b3b2ffce"
server_key_iv: "4b88585baf5d7ca8a2cd878369dc8aa6"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.417094 31077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:38:18.418246 31092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:18.418509 31077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:18.418612 31077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "27fbb243b0324a879895634c6ffa2114"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-3jql"
server_key: "de3ed76247bf96c679f2be35b3b2ffce"
server_key_iv: "4b88585baf5d7ca8a2cd878369dc8aa6"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.418669 31077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:18.421958 31077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:18.424607 31077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:18.424790 31077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:18.429095 31077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40655
I20260620 18:38:18.429185 31144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40655 every 8 connection(s)
I20260620 18:38:18.429472 31077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:18.430366 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31077
I20260620 18:38:18.430495 31145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:18.430534 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:18.430833 10677 external_mini_cluster.cc:1428] Setting key f414fd486d95bcec53d8941f9998d5e4
I20260620 18:38:18.433805 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:18.431476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54347 (local address 127.0.0.1:40655)
0620 18:38:18.431695 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:18.431702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:18.431749 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:38:18.432223 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:18.432231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:18.432259 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:18.432386 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:18.433180 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:18.433184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:18.433184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:18.433189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:18.433238 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:18.433249 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:18.433316 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:18.433608 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:38:18.433655 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260620 18:38:18.434612 31145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: Bootstrap starting.
I20260620 18:38:18.434878 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:18.435290 31145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:18.435570 31145 log.cc:824] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:18.436306 31145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: No bootstrap required, opened a new log
I20260620 18:38:18.438073 31145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260620 18:38:18.438464 31145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:18.438539 31145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fbb243b0324a879895634c6ffa2114, State: Initialized, Role: FOLLOWER
I20260620 18:38:18.438669 31145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260620 18:38:18.438807 31145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:18.438853 31145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:18.438902 31145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:18.439697 31145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260620 18:38:18.439800 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:18.436365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57353 (local address 127.0.0.1:40655)
0620 18:38:18.436434 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:18.436436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:18.438906 (+ 2470us) server_negotiation.cc:407] Connection header received
0620 18:38:18.438927 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:18.438928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:18.438932 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:18.438980 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:18.439522 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:18.439523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:18.439523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:18.439527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:18.439539 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:18.439542 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:18.439593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:18.439677 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:38:18.439688 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:18.439808 31145 leader_election.cc:302] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27fbb243b0324a879895634c6ffa2114; no voters:
I20260620 18:38:18.440080 31145 leader_election.cc:288] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:18.440232 31155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:18.440412 31155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [term 1 LEADER]: Becoming Leader. State: Replica: 27fbb243b0324a879895634c6ffa2114, State: Running, Role: LEADER
I20260620 18:38:18.440529 31155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260620 18:38:18.440573 31145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:18.441169 31157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27fbb243b0324a879895634c6ffa2114. Latest consensus state: current_term: 1 leader_uuid: "27fbb243b0324a879895634c6ffa2114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260620 18:38:18.441226 31156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27fbb243b0324a879895634c6ffa2114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fbb243b0324a879895634c6ffa2114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260620 18:38:18.441303 31157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:18.441322 31156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:18.442836 31171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:18.442972 31171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:18.443044 31163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:18.443290 31163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:18.445245 31163 catalog_manager.cc:1347] Generated new cluster ID: 1b072bbc7ccb4942805eb0e3afd85f2e
I20260620 18:38:18.445327 31163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:18.457795 31163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:18.457973 31163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:18.463114 31163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27fbb243b0324a879895634c6ffa2114: Generated new TSK 0
I20260620 18:38:18.463292 31163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:18.465158 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31077
2026-06-20T18:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:18 dist-test-slave-3jql krb5kdc[31176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:18 dist-test-slave-3jql krb5kdc[31176](info): set up 2 sockets
Jun 20 18:38:18 dist-test-slave-3jql krb5kdc[31176](info): commencing operation
krb5kdc: starting...
W20260620 18:38:19.546811 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:19.657811 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:19.736117 31189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:19.736334 31189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:19.736370 31189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:19.737473 31189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:19.737532 31189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:19.737557 31189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:19.737583 31189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:19.737612 31189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:19.739074 31189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:19.739331 31189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:19.739542 31189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:19.744380 31195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:19.744390 31194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:19.744477 31197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:19.744612 31189 server_base.cc:1034] running on GCE node
I20260620 18:38:19.744822 31189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:19.745049 31189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:19.746276 31189 hybrid_clock.cc:648] HybridClock initialized: now 1781980699746262 us; error 38 us; skew 500 ppm
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:19.748766 31189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:19.749171 31189 webserver.cc:460] Webserver started at http://127.0.0.1:39501/ using document root <none> and password file <none>
I20260620 18:38:19.749395 31189 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:19.749457 31189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:19.749596 31189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:19.750633 31189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "eea700d266b84d6f9391bcd321025bba"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-3jql"
server_key: "eec52d0c96706e66cdea11ae3b795211"
server_key_iv: "cddbb53503c0bd9619dd2900993cbfed"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.751044 31189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "eea700d266b84d6f9391bcd321025bba"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-3jql"
server_key: "eec52d0c96706e66cdea11ae3b795211"
server_key_iv: "cddbb53503c0bd9619dd2900993cbfed"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.752663 31189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:19.753432 31204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.753752 31189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:19.753850 31189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "eea700d266b84d6f9391bcd321025bba"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-3jql"
server_key: "eec52d0c96706e66cdea11ae3b795211"
server_key_iv: "cddbb53503c0bd9619dd2900993cbfed"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.753934 31189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:19.769450 31189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:19.771423 31189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:19.771580 31189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:19.775525 31189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32815
I20260620 18:38:19.775560 31256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32815 every 8 connection(s)
I20260620 18:38:19.775913 31189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:19.776748 31257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:19.778952 31257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: Bootstrap starting.
I20260620 18:38:19.779464 31257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:19.779693 31257 log.cc:824] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:19.780405 31257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: No bootstrap required, opened a new log
I20260620 18:38:19.781967 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31189
I20260620 18:38:19.782066 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:19.782090 31257 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } }
I20260620 18:38:19.782261 10677 external_mini_cluster.cc:1428] Setting key c4ef0726bc5a444ce7c03b841153783b
I20260620 18:38:19.782279 31257 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:19.782351 31257 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea700d266b84d6f9391bcd321025bba, State: Initialized, Role: FOLLOWER
I20260620 18:38:19.782505 31257 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } }
I20260620 18:38:19.782603 31257 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:19.782646 31257 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:19.783036 31257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:19.784271 31257 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } }
I20260620 18:38:19.784380 31257 leader_election.cc:302] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea700d266b84d6f9391bcd321025bba; no voters:
I20260620 18:38:19.784564 31257 leader_election.cc:288] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:19.784596 31262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:19.784729 31262 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [term 1 LEADER]: Becoming Leader. State: Replica: eea700d266b84d6f9391bcd321025bba, State: Running, Role: LEADER
I20260620 18:38:19.784834 31257 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:19.784827 31262 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } }
I20260620 18:38:19.785919 31263 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea700d266b84d6f9391bcd321025bba. Latest consensus state: current_term: 1 leader_uuid: "eea700d266b84d6f9391bcd321025bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } } }
I20260620 18:38:19.786013 31263 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:19.786396 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea700d266b84d6f9391bcd321025bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea700d266b84d6f9391bcd321025bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32815 } } }
I20260620 18:38:19.786484 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:19.786852 31278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:19.786913 31278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:19.786974 31273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:19.787117 31273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:19.789620 31273 catalog_manager.cc:1347] Generated new cluster ID: 3324e2e0273841ba96f5cc5fb7183ea4
I20260620 18:38:19.789676 31273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:38:19.794513 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.782786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58415 (local address 127.0.0.1:32815)
0620 18:38:19.782989 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:19.782996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:19.783042 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:19.783195 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:19.783200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:19.783241 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:19.783345 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:19.784248 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.785666 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.786774 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.787370 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.787498 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.790457 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:19.790461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:19.790462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:19.790472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:19.792471 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.792871 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.792873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.792874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.793416 (+ 542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.793641 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.793642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.793642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.793932 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:19.794098 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:19.794196 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:38:19.794324 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
I20260620 18:38:19.795475 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:19.798601 31273 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:19.799296 31273 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:19.802249 31273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea700d266b84d6f9391bcd321025bba: Generated new TSK 0
I20260620 18:38:19.802394 31273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:19.803964 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.796862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52761 (local address 127.0.0.1:32815)
0620 18:38:19.796921 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:19.796922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:19.797112 (+ 190us) server_negotiation.cc:407] Connection header received
0620 18:38:19.797175 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:19.797176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:19.797186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:19.797252 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:19.797889 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.798481 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.799486 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.799819 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.799890 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.801228 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:19.801229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:19.801229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:19.801236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:19.802895 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.803240 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.803241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.803242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.803297 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.803469 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.803470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.803470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.803593 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:19.803696 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:19.803790 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:38:19.803877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:38:19.805946 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31189
2026-06-20T18:38:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31288](info): set up 2 sockets
Jun 20 18:38:19 dist-test-slave-3jql krb5kdc[31288](info): commencing operation
krb5kdc: starting...
W20260620 18:38:20.839092 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:20 dist-test-slave-3jql krb5kdc[31288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:20.961576 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:21.039134 31301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:21.039354 31301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:21.039390 31301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:21.040488 31301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:21.040552 31301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:21.040575 31301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:21.040597 31301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:21.040627 31301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:21.042078 31301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:21.042335 31301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:21.042543 31301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:21.047467 31306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:21.047564 31307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:21.047468 31309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:21.047797 31301 server_base.cc:1034] running on GCE node
I20260620 18:38:21.048014 31301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:21.048251 31301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:21.049404 31301 hybrid_clock.cc:648] HybridClock initialized: now 1781980701049390 us; error 27 us; skew 500 ppm
Jun 20 18:38:21 dist-test-slave-3jql krb5kdc[31288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:21.051805 31301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:21.052234 31301 webserver.cc:460] Webserver started at http://127.0.0.1:40999/ using document root <none> and password file <none>
I20260620 18:38:21.052444 31301 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:21.052512 31301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:21.052641 31301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:21.053620 31301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7b2d641bd4a040cf8b0174232202bb18"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-3jql"
server_key: "ce1a929798d420582b1ce2ceb33476bb"
server_key_iv: "3e2ff9f333325d608f37cf7d4f6396b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.053958 31301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7b2d641bd4a040cf8b0174232202bb18"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-3jql"
server_key: "ce1a929798d420582b1ce2ceb33476bb"
server_key_iv: "3e2ff9f333325d608f37cf7d4f6396b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.055307 31301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:21.056046 31316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:21.056237 31301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:21.056334 31301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7b2d641bd4a040cf8b0174232202bb18"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-3jql"
server_key: "ce1a929798d420582b1ce2ceb33476bb"
server_key_iv: "3e2ff9f333325d608f37cf7d4f6396b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.056411 31301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:21.066256 31301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:21.068826 31301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:21.068940 31301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:21.072901 31301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45595
I20260620 18:38:21.072961 31368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45595 every 8 connection(s)
I20260620 18:38:21.073278 31301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:21.074085 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:21.075624 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31301
I20260620 18:38:21.075762 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:21.075969 10677 external_mini_cluster.cc:1428] Setting key e430b8bdb2fe0a720136c8e4991e5c91
I20260620 18:38:21.076532 31369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: Bootstrap starting.
I20260620 18:38:21.077214 31369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:21.077525 31369 log.cc:824] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:21.078176 31369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: No bootstrap required, opened a new log
I20260620 18:38:21.079965 31369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260620 18:38:21.080113 31369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:21.080170 31369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2d641bd4a040cf8b0174232202bb18, State: Initialized, Role: FOLLOWER
I20260620 18:38:21.080557 31369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260620 18:38:21.080648 31369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:21.080714 31369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:21.080756 31369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:21.081337 31369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260620 18:38:21.081359 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:21.076486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37611 (local address 127.0.0.1:45595)
0620 18:38:21.076662 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:21.076671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:21.076704 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:38:21.077298 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:21.077307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:21.077375 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:21.077536 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:21.077809 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.078804 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.079545 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.079939 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.080063 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:21.080657 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:21.080661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:21.080661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:21.080891 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:21.080924 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:21.080935 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:21.081047 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:21.081127 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:38:21.081202 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260620 18:38:21.081420 31369 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2d641bd4a040cf8b0174232202bb18; no voters:
I20260620 18:38:21.081578 31369 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:21.081627 31374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:21.081792 31374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2d641bd4a040cf8b0174232202bb18, State: Running, Role: LEADER
I20260620 18:38:21.081820 31369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:21.081924 31374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20260620 18:38:21.082222 31374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2d641bd4a040cf8b0174232202bb18. Latest consensus state: current_term: 1 leader_uuid: "7b2d641bd4a040cf8b0174232202bb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20260620 18:38:21.082232 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2d641bd4a040cf8b0174232202bb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2d641bd4a040cf8b0174232202bb18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20260620 18:38:21.082319 31374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:21.082336 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:21.082345 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:21.083699 31390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:21.083761 31390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:21.083833 31381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:21.084004 31381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:21.085562 31381 catalog_manager.cc:1347] Generated new cluster ID: 09839c02e48e40e5a8a86fce9109d0ac
I20260620 18:38:21.085619 31381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:21.087092 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:21.084000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37387 (local address 127.0.0.1:45595)
0620 18:38:21.084035 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:21.084037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:21.084158 (+ 121us) server_negotiation.cc:407] Connection header received
0620 18:38:21.084218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:21.084219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:21.084230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:21.084292 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:21.084437 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.085152 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.085949 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.086164 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.086245 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:21.086854 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:21.086855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:21.086856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:21.086860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:21.086872 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:21.086876 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:21.086929 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:21.086998 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:38:21.087018 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:21.103156 31381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:21.103806 31381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:21.116521 31381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2d641bd4a040cf8b0174232202bb18: Generated new TSK 0
I20260620 18:38:21.116689 31381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:21.124014 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31301
2026-06-20T18:38:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:21 dist-test-slave-3jql krb5kdc[31400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:21 dist-test-slave-3jql krb5kdc[31400](info): set up 2 sockets
Jun 20 18:38:21 dist-test-slave-3jql krb5kdc[31400](info): commencing operation
krb5kdc: starting...
W20260620 18:38:22.158836 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:22 dist-test-slave-3jql krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:22.273473 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:22.351495 31413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:22.351720 31413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:22.351758 31413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:22.352913 31413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:22.352982 31413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:22.353006 31413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:22.353034 31413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:22.353065 31413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:22.354489 31413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:22.354812 31413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:22.355031 31413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:22.360136 31419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:22.360180 31413 server_base.cc:1034] running on GCE node
W20260620 18:38:22.360144 31418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:22.360163 31421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:22.360561 31413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:22.360841 31413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:22.361996 31413 hybrid_clock.cc:648] HybridClock initialized: now 1781980702361985 us; error 31 us; skew 500 ppm
Jun 20 18:38:22 dist-test-slave-3jql krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:22.369010 31413 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:22.369460 31413 webserver.cc:460] Webserver started at http://127.0.0.1:42421/ using document root <none> and password file <none>
I20260620 18:38:22.369689 31413 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:22.369753 31413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:22.369892 31413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:22.370982 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7a1db253833846d293c129f70b2f1b1e"
format_stamp: "Formatted at 2026-06-20 18:38:22 on dist-test-slave-3jql"
server_key: "3a5e1f6eee9f0631aa393f93513e06f5"
server_key_iv: "2da35e891911d5627b3ba913d0ec47c3"
server_key_version: "encryptionkey@0"
I20260620 18:38:22.371313 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7a1db253833846d293c129f70b2f1b1e"
format_stamp: "Formatted at 2026-06-20 18:38:22 on dist-test-slave-3jql"
server_key: "3a5e1f6eee9f0631aa393f93513e06f5"
server_key_iv: "2da35e891911d5627b3ba913d0ec47c3"
server_key_version: "encryptionkey@0"
I20260620 18:38:22.372581 31413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:22.373302 31428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:22.373476 31413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:22.373585 31413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7a1db253833846d293c129f70b2f1b1e"
format_stamp: "Formatted at 2026-06-20 18:38:22 on dist-test-slave-3jql"
server_key: "3a5e1f6eee9f0631aa393f93513e06f5"
server_key_iv: "2da35e891911d5627b3ba913d0ec47c3"
server_key_version: "encryptionkey@0"
I20260620 18:38:22.373667 31413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:22.376927 31413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:22.378713 31413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:22.378880 31413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:22.382826 31413 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42801
I20260620 18:38:22.382855 31480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42801 every 8 connection(s)
I20260620 18:38:22.383162 31413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:22.384048 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:22.386044 31481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: Bootstrap starting.
I20260620 18:38:22.386535 31481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:22.386809 31481 log.cc:824] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:22.387420 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31413
I20260620 18:38:22.387428 31481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: No bootstrap required, opened a new log
I20260620 18:38:22.387589 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:22.387775 10677 external_mini_cluster.cc:1428] Setting key 10743544c4b52c1b801315b97b142cdf
I20260620 18:38:22.389569 31481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 18:38:22.389786 31481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:22.389819 31481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1db253833846d293c129f70b2f1b1e, State: Initialized, Role: FOLLOWER
I20260620 18:38:22.389952 31481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 18:38:22.390038 31481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:22.390118 31481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:22.390183 31481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:22.390797 31481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 18:38:22.390918 31481 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a1db253833846d293c129f70b2f1b1e; no voters:
I20260620 18:38:22.391059 31481 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:22.391150 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:22.388226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53255 (local address 127.0.0.1:42801)
0620 18:38:22.388417 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:22.388426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:22.388457 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:38:22.389760 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:22.389768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:22.389801 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:22.389941 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:22.390590 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:22.390594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:22.390594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:22.390598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:22.390642 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:22.390656 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:22.390732 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:22.390900 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:38:22.390941 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260620 18:38:22.391131 31486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:22.391299 31481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:22.391443 31486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [term 1 LEADER]: Becoming Leader. State: Replica: 7a1db253833846d293c129f70b2f1b1e, State: Running, Role: LEADER
I20260620 18:38:22.391543 31486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 18:38:22.391878 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1db253833846d293c129f70b2f1b1e. Latest consensus state: current_term: 1 leader_uuid: "7a1db253833846d293c129f70b2f1b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } } }
I20260620 18:38:22.391955 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:22.392091 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:22.393025 31487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a1db253833846d293c129f70b2f1b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1db253833846d293c129f70b2f1b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } } }
I20260620 18:38:22.393105 31487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:22.393106 31502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:22.393211 31502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:22.393287 31495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:22.393451 31495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:22.394688 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:22.393646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35973 (local address 127.0.0.1:42801)
0620 18:38:22.393686 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:22.393688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:22.393805 (+ 117us) server_negotiation.cc:407] Connection header received
0620 18:38:22.393865 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:22.393866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:22.393870 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:22.393928 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:22.394473 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:22.394474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:22.394474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:22.394478 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:22.394491 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:22.394495 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:22.394546 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:22.394617 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:38:22.394628 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:38:22.395110 31495 catalog_manager.cc:1347] Generated new cluster ID: 41716f0e073045c596d31cb278b70936
I20260620 18:38:22.395161 31495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:22.414953 31495 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:22.415110 31495 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:22.419445 31495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a1db253833846d293c129f70b2f1b1e: Generated new TSK 0
I20260620 18:38:22.419661 31495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:22.425784 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31413
2026-06-20T18:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:22 dist-test-slave-3jql krb5kdc[31512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:22 dist-test-slave-3jql krb5kdc[31512](info): set up 2 sockets
Jun 20 18:38:22 dist-test-slave-3jql krb5kdc[31512](info): commencing operation
krb5kdc: starting...
W20260620 18:38:23.467052 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:23.594061 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:23.678447 31525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:23.678704 31525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:23.678728 31525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:23.679934 31525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:23.679981 31525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:23.679997 31525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:23.680009 31525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:23.680061 31525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:23.681548 31525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:23.681773 31525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:23.682049 31525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:23.686858 31530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.686827 31531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.686846 31533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:23.686899 31525 server_base.cc:1034] running on GCE node
I20260620 18:38:23.687265 31525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:23.687554 31525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:23.688690 31525 hybrid_clock.cc:648] HybridClock initialized: now 1781980703688679 us; error 33 us; skew 500 ppm
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:23.691282 31525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:23.691738 31525 webserver.cc:460] Webserver started at http://127.0.0.1:43519/ using document root <none> and password file <none>
I20260620 18:38:23.691972 31525 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:23.692045 31525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:23.692191 31525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:23.693315 31525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "d25d7c244dda4157ae22336d8c2ffc1d"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-3jql"
server_key: "0f734afe3ca6a987d095f653fe10514b"
server_key_iv: "6bba76929f98ce233b9c481a0dc07207"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.693706 31525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "d25d7c244dda4157ae22336d8c2ffc1d"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-3jql"
server_key: "0f734afe3ca6a987d095f653fe10514b"
server_key_iv: "6bba76929f98ce233b9c481a0dc07207"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.695549 31525 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:38:23.696396 31540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:23.696583 31525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:23.696681 31525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "d25d7c244dda4157ae22336d8c2ffc1d"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-3jql"
server_key: "0f734afe3ca6a987d095f653fe10514b"
server_key_iv: "6bba76929f98ce233b9c481a0dc07207"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.696762 31525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:23.728335 31525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:23.730270 31525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:23.730440 31525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:23.734719 31525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37215
I20260620 18:38:23.734814 31592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37215 every 8 connection(s)
I20260620 18:38:23.735174 31525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:23.735988 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:23.738135 31593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d: Bootstrap starting.
I20260620 18:38:23.738778 31593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:23.739068 31593 log.cc:824] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:23.739445 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31525
I20260620 18:38:23.739583 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:23.739779 10677 external_mini_cluster.cc:1428] Setting key 255960d4168c83adfabfdc79d43a7b61
I20260620 18:38:23.739904 31593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d: No bootstrap required, opened a new log
I20260620 18:38:23.741763 31593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } }
I20260620 18:38:23.741894 31593 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:23.741953 31593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25d7c244dda4157ae22336d8c2ffc1d, State: Initialized, Role: FOLLOWER
I20260620 18:38:23.742084 31593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } }
I20260620 18:38:23.742164 31593 raft_consensus.cc:388] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:23.742209 31593 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:23.742255 31593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:23.742843 31593 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } }
I20260620 18:38:23.742964 31593 leader_election.cc:302] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d25d7c244dda4157ae22336d8c2ffc1d; no voters:
I20260620 18:38:23.743119 31593 leader_election.cc:288] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:23.743144 31598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:23.743309 31598 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [term 1 LEADER]: Becoming Leader. State: Replica: d25d7c244dda4157ae22336d8c2ffc1d, State: Running, Role: LEADER
I20260620 18:38:23.743376 31593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:23.743433 31598 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } }
I20260620 18:38:23.743753 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25d7c244dda4157ae22336d8c2ffc1d. Latest consensus state: current_term: 1 leader_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } } }
I20260620 18:38:23.743758 31600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d7c244dda4157ae22336d8c2ffc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37215 } } }
I20260620 18:38:23.743916 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:23.743928 31600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:23.744166 31606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:23.744659 31606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:23.746367 31606 catalog_manager.cc:1347] Generated new cluster ID: ec10325140ea4ceb989115536e55cd86
I20260620 18:38:23.746415 31606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:38:23.751693 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:23.740319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:38439 (local address 127.0.0.1:37215)
0620 18:38:23.740503 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:23.740514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:23.740542 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:38:23.741123 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:23.741130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:23.741176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:23.741319 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:23.741919 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.743464 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.744580 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.745166 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.745277 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:23.747997 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:23.748001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:23.748001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:23.748012 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:23.749901 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.750333 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.750336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.750336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.750443 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.750679 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.750680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.750680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.751032 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:23.751205 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:23.751284 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:38:23.751383 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260620 18:38:23.752668 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:23.753427 31606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:23.754093 31606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:23.756431 31606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25d7c244dda4157ae22336d8c2ffc1d: Generated new TSK 0
I20260620 18:38:23.756568 31606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:23.760820 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:23.754119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:59371 (local address 127.0.0.1:37215)
0620 18:38:23.754158 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:23.754160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:23.754301 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:38:23.754346 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:23.754347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:23.754356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:23.754423 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:23.755011 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.755753 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.756729 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.757058 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.757120 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:23.758225 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:23.758226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:23.758227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:23.758233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:23.759768 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.760103 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.760104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.760105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.760163 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.760366 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.760367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.760367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.760485 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:23.760582 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:23.760673 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 18:38:23.760733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 18:38:23.762444 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31525
W20260620 18:38:23.766583 31510 connection.cc:535] client connection to 127.0.0.1:37215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37215): Connection reset by peer (error 104)
2026-06-20T18:38:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31624](info): set up 2 sockets
Jun 20 18:38:23 dist-test-slave-3jql krb5kdc[31624](info): commencing operation
krb5kdc: starting...
W20260620 18:38:24.790954 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:24 dist-test-slave-3jql krb5kdc[31624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:24.909731 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:24.988113 31637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:24.988293 31637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:24.988317 31637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:24.989413 31637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:24.989451 31637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:24.989465 31637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:24.989478 31637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:24.989531 31637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:24.990931 31637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:24.991153 31637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:24.991371 31637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:24.995918 31642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.995970 31645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.995991 31643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:24.996217 31637 server_base.cc:1034] running on GCE node
I20260620 18:38:24.996481 31637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:24.996753 31637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:24.997936 31637 hybrid_clock.cc:648] HybridClock initialized: now 1781980704997929 us; error 38 us; skew 500 ppm
Jun 20 18:38:24 dist-test-slave-3jql krb5kdc[31624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:25.000319 31637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:25.000674 31637 webserver.cc:460] Webserver started at http://127.0.0.1:39793/ using document root <none> and password file <none>
I20260620 18:38:25.000833 31637 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:25.000870 31637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:25.000952 31637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:25.001852 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "e25df745f1704f729db1a20894534bf5"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-3jql"
server_key: "d4e355e52825ca2c767d200e06b25dce"
server_key_iv: "8daa437ef0d2ceedfd2c4b8472d6fa36"
server_key_version: "encryptionkey@0"
I20260620 18:38:25.002127 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "e25df745f1704f729db1a20894534bf5"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-3jql"
server_key: "d4e355e52825ca2c767d200e06b25dce"
server_key_iv: "8daa437ef0d2ceedfd2c4b8472d6fa36"
server_key_version: "encryptionkey@0"
I20260620 18:38:25.003623 31637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:25.004288 31652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:25.004487 31637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:25.004566 31637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "e25df745f1704f729db1a20894534bf5"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-3jql"
server_key: "d4e355e52825ca2c767d200e06b25dce"
server_key_iv: "8daa437ef0d2ceedfd2c4b8472d6fa36"
server_key_version: "encryptionkey@0"
I20260620 18:38:25.004617 31637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:25.012466 31637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:25.014550 31637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:25.014650 31637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:25.018190 31637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33681
I20260620 18:38:25.018258 31704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33681 every 8 connection(s)
I20260620 18:38:25.018565 31637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:25.019371 31705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:25.021370 31705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: Bootstrap starting.
I20260620 18:38:25.021833 31705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:25.022061 31705 log.cc:824] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:25.022722 31705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: No bootstrap required, opened a new log
I20260620 18:38:25.023923 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31637
I20260620 18:38:25.024070 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:25.024251 10677 external_mini_cluster.cc:1428] Setting key fec97fcf020fe0065c570a242c9877e4
I20260620 18:38:25.024420 31705 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260620 18:38:25.024569 31705 raft_consensus.cc:374] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:25.024605 31705 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25df745f1704f729db1a20894534bf5, State: Initialized, Role: FOLLOWER
I20260620 18:38:25.024793 31705 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260620 18:38:25.024897 31705 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:25.024945 31705 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:25.024995 31705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:25.025812 31705 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260620 18:38:25.025933 31705 leader_election.cc:302] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25df745f1704f729db1a20894534bf5; no voters:
I20260620 18:38:25.026134 31705 leader_election.cc:288] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:25.026165 31710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:25.026262 31710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [term 1 LEADER]: Becoming Leader. State: Replica: e25df745f1704f729db1a20894534bf5, State: Running, Role: LEADER
I20260620 18:38:25.026401 31705 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:25.026392 31710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260620 18:38:25.026886 31711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e25df745f1704f729db1a20894534bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } } }
I20260620 18:38:25.027118 31711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:25.027051 31712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25df745f1704f729db1a20894534bf5. Latest consensus state: current_term: 1 leader_uuid: "e25df745f1704f729db1a20894534bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25df745f1704f729db1a20894534bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } } }
I20260620 18:38:25.027505 31712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:25.028133 31726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:25.028185 31726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:25.028273 31723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:25.028445 31723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:25.029966 31723 catalog_manager.cc:1347] Generated new cluster ID: 4394ebd9e71b40b98c48c28cfffc070a
I20260620 18:38:25.030018 31723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:25.030051 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:25.024716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:33621 (local address 127.0.0.1:33681)
0620 18:38:25.024888 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:25.024897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:25.024928 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:38:25.025133 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:25.025140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:25.025192 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:25.025322 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:25.025496 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:25.026581 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:25.027794 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:25.028379 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:25.028509 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:25.029311 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:25.029315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:25.029315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:25.029546 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:25.029585 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:25.029600 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:25.029692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:25.029799 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 18:38:25.029825 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260620 18:38:25.030929 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:25.034969 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:25.032182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:47917 (local address 127.0.0.1:33681)
0620 18:38:25.032216 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:25.032217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:25.032324 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:38:25.032377 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:25.032378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:25.032388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:25.032443 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:25.032594 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:25.033065 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:25.033771 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:25.033978 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:25.034059 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:25.034680 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:25.034682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:25.034682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:25.034686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:25.034698 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:25.034702 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:25.034773 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:25.034846 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:38:25.034858 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:25.054690 31723 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:25.055332 31723 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:25.059700 31723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e25df745f1704f729db1a20894534bf5: Generated new TSK 0
I20260620 18:38:25.059887 31723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:25.063915 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31637
W20260620 18:38:25.068208 31622 connection.cc:535] client connection to 127.0.0.1:33681 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33681): Connection reset by peer (error 104)
2026-06-20T18:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:25 dist-test-slave-3jql krb5kdc[31736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:25 dist-test-slave-3jql krb5kdc[31736](info): set up 2 sockets
Jun 20 18:38:25 dist-test-slave-3jql krb5kdc[31736](info): commencing operation
krb5kdc: starting...
W20260620 18:38:26.094955 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:26 dist-test-slave-3jql krb5kdc[31736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:26.211516 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:26.289520 31749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:26.289753 31749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:26.289793 31749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:26.290953 31749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:26.291028 31749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:26.291052 31749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:26.291085 31749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:26.291116 31749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:26.292572 31749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:26.292840 31749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:26.293067 31749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:26.297853 31757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:26.297816 31755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:26.297823 31754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:26.298089 31749 server_base.cc:1034] running on GCE node
I20260620 18:38:26.298321 31749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:26.298588 31749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:26.299731 31749 hybrid_clock.cc:648] HybridClock initialized: now 1781980706299714 us; error 31 us; skew 500 ppm
Jun 20 18:38:26 dist-test-slave-3jql krb5kdc[31736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:26.302220 31749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:26.302687 31749 webserver.cc:460] Webserver started at http://127.0.0.1:36803/ using document root <none> and password file <none>
I20260620 18:38:26.302947 31749 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:26.303025 31749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:26.303176 31749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:26.304224 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "490ade38de4e4905a14c59eb50c4d39a"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-3jql"
server_key: "92f8c3a104b68f5e3429280c371c88b1"
server_key_iv: "512aea47b4d99255f2e408b09ea23db3"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.304575 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "490ade38de4e4905a14c59eb50c4d39a"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-3jql"
server_key: "92f8c3a104b68f5e3429280c371c88b1"
server_key_iv: "512aea47b4d99255f2e408b09ea23db3"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.305987 31749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:26.306718 31764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:26.307039 31749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:26.307128 31749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "490ade38de4e4905a14c59eb50c4d39a"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-3jql"
server_key: "92f8c3a104b68f5e3429280c371c88b1"
server_key_iv: "512aea47b4d99255f2e408b09ea23db3"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.307210 31749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:26.311038 31749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:26.313494 31749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:26.313668 31749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:26.317886 31749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46459
I20260620 18:38:26.317914 31816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46459 every 8 connection(s)
I20260620 18:38:26.318287 31749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:26.318960 31817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:26.321069 31817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a: Bootstrap starting.
I20260620 18:38:26.321605 31817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:26.321871 31817 log.cc:824] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:26.322548 31817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a: No bootstrap required, opened a new log
I20260620 18:38:26.324110 31817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } }
I20260620 18:38:26.324235 31817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:26.324257 31817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490ade38de4e4905a14c59eb50c4d39a, State: Initialized, Role: FOLLOWER
I20260620 18:38:26.324433 31817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } }
I20260620 18:38:26.324496 31817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:26.324515 31817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:26.324568 31817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:26.325090 31817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } }
I20260620 18:38:26.325222 31817 leader_election.cc:302] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 490ade38de4e4905a14c59eb50c4d39a; no voters:
I20260620 18:38:26.325332 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31749
I20260620 18:38:26.325376 31817 leader_election.cc:288] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:26.325457 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:26.325451 31820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:26.325596 31820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [term 1 LEADER]: Becoming Leader. State: Replica: 490ade38de4e4905a14c59eb50c4d39a, State: Running, Role: LEADER
I20260620 18:38:26.325621 10677 external_mini_cluster.cc:1428] Setting key b8d2e98b2e9ca5741e0302261d36a29b
I20260620 18:38:26.325620 31817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:26.325675 31820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } }
I20260620 18:38:26.326014 31820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490ade38de4e4905a14c59eb50c4d39a. Latest consensus state: current_term: 1 leader_uuid: "490ade38de4e4905a14c59eb50c4d39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } } }
I20260620 18:38:26.326109 31820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:26.326357 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "490ade38de4e4905a14c59eb50c4d39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490ade38de4e4905a14c59eb50c4d39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46459 } } }
I20260620 18:38:26.326494 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:26.327320 31829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:26.327761 31829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:26.328122 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:26.326203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:38929 (local address 127.0.0.1:46459)
0620 18:38:26.326593 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:26.326604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:26.326638 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:26.326795 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:26.326803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:26.326836 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:26.326986 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:26.327695 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:26.327699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:26.327699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:26.327703 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:26.327742 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:26.327757 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:26.327834 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:26.327914 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:38:26.327955 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":87,"threads_started":1}
I20260620 18:38:26.329059 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:26.330157 31829 catalog_manager.cc:1347] Generated new cluster ID: b884b3145191462c87343b3bf9b17c41
I20260620 18:38:26.330220 31829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:26.331385 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:26.330438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:53883 (local address 127.0.0.1:46459)
0620 18:38:26.330483 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:26.330485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:26.330493 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:26.330515 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:26.330516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:26.330521 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:26.330586 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:26.331169 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:26.331170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:26.331171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:26.331175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:26.331190 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:26.331195 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:26.331262 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:26.331308 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 18:38:26.331321 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 18:38:26.335515 31829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:26.335650 31829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:26.341326 31829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 490ade38de4e4905a14c59eb50c4d39a: Generated new TSK 0
I20260620 18:38:26.341492 31829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:26.344015 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31749
2026-06-20T18:38:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:26 dist-test-slave-3jql krb5kdc[31848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:26 dist-test-slave-3jql krb5kdc[31848](info): set up 2 sockets
Jun 20 18:38:26 dist-test-slave-3jql krb5kdc[31848](info): commencing operation
krb5kdc: starting...
W20260620 18:38:27.390995 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:27.505924 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:27.591758 31861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:27.591992 31861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:27.592033 31861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:27.593264 31861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:27.593336 31861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:27.593366 31861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:27.593400 31861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:27.593427 31861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:27.595014 31861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:27.595295 31861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:27.595534 31861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:27.600600 31866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:27.600612 31867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:27.600818 31861 server_base.cc:1034] running on GCE node
W20260620 18:38:27.600606 31869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:27.601141 31861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:27.601413 31861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:27.602581 31861 hybrid_clock.cc:648] HybridClock initialized: now 1781980707602551 us; error 38 us; skew 500 ppm
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:27.611140 31861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:27.611650 31861 webserver.cc:460] Webserver started at http://127.0.0.1:43895/ using document root <none> and password file <none>
I20260620 18:38:27.611891 31861 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:27.611965 31861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:27.612126 31861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:27.613210 31861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7f8c677fd7f64d9386a7571252aeec6c"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-3jql"
server_key: "6265fc80fa2a8a01374b6509989fe9aa"
server_key_iv: "e7d111c545446872814a7de7cc857150"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.613590 31861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7f8c677fd7f64d9386a7571252aeec6c"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-3jql"
server_key: "6265fc80fa2a8a01374b6509989fe9aa"
server_key_iv: "e7d111c545446872814a7de7cc857150"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.615046 31861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:27.615805 31876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.615986 31861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:27.616082 31861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7f8c677fd7f64d9386a7571252aeec6c"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-3jql"
server_key: "6265fc80fa2a8a01374b6509989fe9aa"
server_key_iv: "e7d111c545446872814a7de7cc857150"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.616170 31861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:27.627187 31861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:27.628890 31861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:27.629041 31861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:27.632962 31861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37487
I20260620 18:38:27.632993 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37487 every 8 connection(s)
I20260620 18:38:27.633308 31861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:27.634127 31929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:27.636308 31929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c: Bootstrap starting.
I20260620 18:38:27.636855 31929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:27.637136 31929 log.cc:824] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:27.637773 31929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c: No bootstrap required, opened a new log
I20260620 18:38:27.639407 31929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260620 18:38:27.639530 31929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:27.639551 31929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8c677fd7f64d9386a7571252aeec6c, State: Initialized, Role: FOLLOWER
I20260620 18:38:27.639714 31929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260620 18:38:27.639770 31929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:27.639809 31929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:27.639870 31929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:27.640365 31929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260620 18:38:27.640446 31929 leader_election.cc:302] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f8c677fd7f64d9386a7571252aeec6c; no voters:
I20260620 18:38:27.640609 31929 leader_election.cc:288] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:27.640610 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31861
I20260620 18:38:27.640650 31932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:27.640729 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:27.640769 31932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [term 1 LEADER]: Becoming Leader. State: Replica: 7f8c677fd7f64d9386a7571252aeec6c, State: Running, Role: LEADER
I20260620 18:38:27.640844 31932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260620 18:38:27.640950 10677 external_mini_cluster.cc:1428] Setting key 484fd6aad000a02b1d614f23b2b5c380
I20260620 18:38:27.640852 31929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:27.641253 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f8c677fd7f64d9386a7571252aeec6c. Latest consensus state: current_term: 1 leader_uuid: "7f8c677fd7f64d9386a7571252aeec6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260620 18:38:27.641400 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:27.641763 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f8c677fd7f64d9386a7571252aeec6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8c677fd7f64d9386a7571252aeec6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260620 18:38:27.641798 31937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:27.641882 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:27.642477 31937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:27.644117 31937 catalog_manager.cc:1347] Generated new cluster ID: 6e3642a1a5284b158c1449477e4cbd0f
I20260620 18:38:27.644172 31937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:38:27.651989 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.641570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:39865 (local address 127.0.0.1:37487)
0620 18:38:27.641933 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:27.641942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:27.642087 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:38:27.642317 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:27.642324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:27.642402 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:27.642530 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:27.643371 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.644661 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.645485 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.645905 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.646029 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.648778 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:27.648782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:27.648783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:27.648793 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:27.650418 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.650854 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.650857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.650857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.650958 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.651194 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.651195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.651195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.651429 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:27.651613 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:27.651705 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:38:27.651813 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":70,"threads_started":1}
I20260620 18:38:27.652532 31937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:27.652855 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:27.653203 31937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:27.658722 31937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f8c677fd7f64d9386a7571252aeec6c: Generated new TSK 0
I20260620 18:38:27.658901 31937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:27.660303 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.654182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:46811 (local address 127.0.0.1:37487)
0620 18:38:27.654217 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:27.654219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:27.654324 (+ 105us) server_negotiation.cc:407] Connection header received
0620 18:38:27.654394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:27.654395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:27.654404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:27.654454 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:27.655028 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.655517 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.656222 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.656445 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.656516 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.657599 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:27.657601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:27.657601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:27.657608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:27.659187 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.659549 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.659550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.659550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.659601 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.659829 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.659830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.659830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.659935 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:27.660038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:27.660158 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 18:38:27.660216 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:27.662074 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31861
W20260620 18:38:27.665983 31846 connection.cc:535] client connection to 127.0.0.1:37487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37487): Connection reset by peer (error 104)
2026-06-20T18:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31960](info): set up 2 sockets
Jun 20 18:38:27 dist-test-slave-3jql krb5kdc[31960](info): commencing operation
krb5kdc: starting...
W20260620 18:38:28.690922 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:28 dist-test-slave-3jql krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:28.805353 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:28.884577 31973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:28.884792 31973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:28.884815 31973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:28.886014 31973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:28.886055 31973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:28.886070 31973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:28.886089 31973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:28.886102 31973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:28.887513 31973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:28.887699 31973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:28.887857 31973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:38:28.892876 31973 server_base.cc:1034] running on GCE node
W20260620 18:38:28.892851 31978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:28.892838 31979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:28.892844 31981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:28.893334 31973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:28.893658 31973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:28.894929 31973 hybrid_clock.cc:648] HybridClock initialized: now 1781980708894890 us; error 42 us; skew 500 ppm
Jun 20 18:38:28 dist-test-slave-3jql krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:28.897773 31973 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:28.898283 31973 webserver.cc:460] Webserver started at http://127.0.0.1:34573/ using document root <none> and password file <none>
I20260620 18:38:28.898528 31973 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:28.898597 31973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:28.898787 31973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:28.899812 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "318436022d91447ba5995396a2c05661"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-3jql"
server_key: "4ba97bb3f72237d8294cd730e4f7cadd"
server_key_iv: "f2657099e3087bfd49550b145870ff77"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.900187 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "318436022d91447ba5995396a2c05661"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-3jql"
server_key: "4ba97bb3f72237d8294cd730e4f7cadd"
server_key_iv: "f2657099e3087bfd49550b145870ff77"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.901487 31973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:38:28.902189 31988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:28.902359 31973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:28.902455 31973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "318436022d91447ba5995396a2c05661"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-3jql"
server_key: "4ba97bb3f72237d8294cd730e4f7cadd"
server_key_iv: "f2657099e3087bfd49550b145870ff77"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.902535 31973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:28.927523 31973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:28.929795 31973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:28.929952 31973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:28.934197 31973 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35449
I20260620 18:38:28.934237 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35449 every 8 connection(s)
I20260620 18:38:28.934595 31973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:28.935503 32041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:28.937703 32041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: Bootstrap starting.
I20260620 18:38:28.938266 32041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:28.938552 32041 log.cc:824] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:28.939291 32041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: No bootstrap required, opened a new log
I20260620 18:38:28.939787 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 31973
I20260620 18:38:28.939931 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:28.940141 10677 external_mini_cluster.cc:1428] Setting key 61835199dd081df20366fd1acedde0f7
I20260620 18:38:28.941398 32041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260620 18:38:28.941555 32041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:28.941622 32041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318436022d91447ba5995396a2c05661, State: Initialized, Role: FOLLOWER
I20260620 18:38:28.941752 32041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260620 18:38:28.941851 32041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:28.941896 32041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:28.941962 32041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:28.942488 32041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260620 18:38:28.942602 32041 leader_election.cc:302] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318436022d91447ba5995396a2c05661; no voters:
I20260620 18:38:28.942783 32041 leader_election.cc:288] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:28.942790 32046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:28.942932 32046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [term 1 LEADER]: Becoming Leader. State: Replica: 318436022d91447ba5995396a2c05661, State: Running, Role: LEADER
I20260620 18:38:28.943053 32041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:28.943065 32046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } }
I20260620 18:38:28.944720 32047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318436022d91447ba5995396a2c05661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } } }
I20260620 18:38:28.944921 32047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:28.944753 32048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318436022d91447ba5995396a2c05661. Latest consensus state: current_term: 1 leader_uuid: "318436022d91447ba5995396a2c05661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318436022d91447ba5995396a2c05661" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35449 } } }
W20260620 18:38:28.945216 32060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 18:38:28.945225 32048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:28.945371 32060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:28.945448 32061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:28.945631 32061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:28.946566 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.940649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:35461 (local address 127.0.0.1:35449)
0620 18:38:28.940822 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.940836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.940860 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:28.941551 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.941558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.941635 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.941782 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.941964 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.943093 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.944286 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.944879 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.944993 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.945863 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.945867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.945867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:28.946069 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.946107 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:28.946121 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.946208 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.946338 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.946364 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260620 18:38:28.947542 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:28.951122 32061 catalog_manager.cc:1347] Generated new cluster ID: fad0e84009584f8ca419acc1822f748f
I20260620 18:38:28.951174 32061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:28.953718 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.948922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:34121 (local address 127.0.0.1:35449)
0620 18:38:28.948982 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.948984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.950939 (+ 1955us) server_negotiation.cc:407] Connection header received
0620 18:38:28.951003 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.951004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.951015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.951064 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.951268 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.951968 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.952635 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.952904 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.952958 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.953539 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.953540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.953540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:28.953544 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.953554 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:28.953557 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.953604 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.953647 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.953655 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:28.956209 32061 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:28.956898 32061 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:28.965476 32061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318436022d91447ba5995396a2c05661: Generated new TSK 0
I20260620 18:38:28.965636 32061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:28.967607 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 31973
W20260620 18:38:28.972051 31958 connection.cc:535] client connection to 127.0.0.1:35449 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35449): Connection reset by peer (error 104)
2026-06-20T18:38:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:29 dist-test-slave-3jql krb5kdc[32073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:29 dist-test-slave-3jql krb5kdc[32073](info): set up 2 sockets
Jun 20 18:38:29 dist-test-slave-3jql krb5kdc[32073](info): commencing operation
krb5kdc: starting...
W20260620 18:38:30.016158 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:30 dist-test-slave-3jql krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:38:30.141181 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:30.222713 32086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:30.222991 32086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:30.223016 32086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:30.224141 32086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:30.224184 32086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:30.224198 32086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:30.224211 32086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:30.224223 32086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:30.225641 32086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:30.225834 32086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:30.225993 32086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:30.230891 32094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.230901 32092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.230993 32091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:30.231122 32086 server_base.cc:1034] running on GCE node
I20260620 18:38:30.231383 32086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:30.231626 32086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:30.232780 32086 hybrid_clock.cc:648] HybridClock initialized: now 1781980710232771 us; error 32 us; skew 500 ppm
Jun 20 18:38:30 dist-test-slave-3jql krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:30.235325 32086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:38:30.235867 32086 webserver.cc:460] Webserver started at http://127.0.0.1:43471/ using document root <none> and password file <none>
I20260620 18:38:30.236138 32086 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:30.236210 32086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:30.236361 32086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:30.237385 32086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "1c7770edb1234e069e4531a85b00b9c4"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-3jql"
server_key: "f3b1aad516d4f3e77d916f4758a85877"
server_key_iv: "72380f2aa562d89638b4ade11db06052"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.237737 32086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "1c7770edb1234e069e4531a85b00b9c4"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-3jql"
server_key: "f3b1aad516d4f3e77d916f4758a85877"
server_key_iv: "72380f2aa562d89638b4ade11db06052"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.239322 32086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:30.240295 32101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:30.240603 32086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:30.240715 32086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "1c7770edb1234e069e4531a85b00b9c4"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-3jql"
server_key: "f3b1aad516d4f3e77d916f4758a85877"
server_key_iv: "72380f2aa562d89638b4ade11db06052"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.240800 32086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:30.244100 32086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:30.246672 32086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:30.246856 32086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:30.251180 32086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36257
I20260620 18:38:30.251224 32153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36257 every 8 connection(s)
I20260620 18:38:30.251545 32086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:30.252593 32154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:30.254913 32154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4: Bootstrap starting.
I20260620 18:38:30.255465 32154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:30.255712 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32086
I20260620 18:38:30.255774 32154 log.cc:824] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:30.255827 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:30.256071 10677 external_mini_cluster.cc:1428] Setting key d99b80ff3cfed9cd57bb456d7282725d
I20260620 18:38:30.256593 32154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4: No bootstrap required, opened a new log
I20260620 18:38:30.258697 32154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260620 18:38:30.258891 32154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:30.258971 32154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7770edb1234e069e4531a85b00b9c4, State: Initialized, Role: FOLLOWER
I20260620 18:38:30.259127 32154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260620 18:38:30.259241 32154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:30.259272 32154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:30.259323 32154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:30.259351 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:30.256567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:37345 (local address 127.0.0.1:36257)
0620 18:38:30.256779 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:30.256788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:30.256825 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:38:30.257814 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:30.257823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:30.257857 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:30.258000 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:30.258697 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:30.258700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:30.258700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:30.258705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:30.258772 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:30.258788 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:30.258859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:30.258949 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 18:38:30.258993 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260620 18:38:30.260157 32154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260620 18:38:30.260411 32154 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7770edb1234e069e4531a85b00b9c4; no voters:
I20260620 18:38:30.260691 32154 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:30.260773 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:30.260721 32159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:30.261045 32159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7770edb1234e069e4531a85b00b9c4, State: Running, Role: LEADER
I20260620 18:38:30.261199 32154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:30.261224 32159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260620 18:38:30.261653 32159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7770edb1234e069e4531a85b00b9c4. Latest consensus state: current_term: 1 leader_uuid: "1c7770edb1234e069e4531a85b00b9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260620 18:38:30.261667 32161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7770edb1234e069e4531a85b00b9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7770edb1234e069e4531a85b00b9c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260620 18:38:30.261914 32161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:30.262161 32159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:30.262365 32172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:30.262918 32172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:30.265069 32172 catalog_manager.cc:1347] Generated new cluster ID: 289404a82d3340b48ddc536f62e2d7e0
I20260620 18:38:30.265141 32172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:30.268384 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:30.267129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:35351 (local address 127.0.0.1:36257)
0620 18:38:30.267178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:30.267181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:30.267282 (+ 101us) server_negotiation.cc:407] Connection header received
0620 18:38:30.267346 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:30.267347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:30.267353 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:30.267446 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:30.268066 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:30.268068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:30.268068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:30.268072 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:30.268087 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:30.268092 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:30.268145 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:30.268296 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:38:30.268323 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:38:30.271057 32172 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:30.271238 32172 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:30.274802 32172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7770edb1234e069e4531a85b00b9c4: Generated new TSK 0
I20260620 18:38:30.274958 32172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:30.281276 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32086
2026-06-20T18:38:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:30 dist-test-slave-3jql krb5kdc[32185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:30 dist-test-slave-3jql krb5kdc[32185](info): set up 2 sockets
Jun 20 18:38:30 dist-test-slave-3jql krb5kdc[32185](info): commencing operation
krb5kdc: starting...
W20260620 18:38:31.323041 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:31.439461 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33101
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:37039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:31.519562 32198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:31.519755 32198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:31.519780 32198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:31.520958 32198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:31.520998 32198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:31.521013 32198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:31.521025 32198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:31.521036 32198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:31.522469 32198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:37039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:33101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:31.522709 32198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:31.522935 32198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:31.527539 32204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:31.527578 32203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:31.527791 32198 server_base.cc:1034] running on GCE node
W20260620 18:38:31.527889 32206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:31.528113 32198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:31.528370 32198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:31.529523 32198 hybrid_clock.cc:648] HybridClock initialized: now 1781980711529519 us; error 37 us; skew 500 ppm
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:31.531858 32198 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:31.532295 32198 webserver.cc:460] Webserver started at http://127.10.109.126:35293/ using document root <none> and password file <none>
I20260620 18:38:31.532508 32198 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:31.532569 32198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:31.532707 32198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:31.533699 32198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "78030cf7c3464dd1aef8cbbd0d5420da"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-3jql"
server_key: "9bc9bcffe046a59616d6f614dbde9c81"
server_key_iv: "7b8b964f7d784e2d7ae7e0fda2cd7994"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.534039 32198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "78030cf7c3464dd1aef8cbbd0d5420da"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-3jql"
server_key: "9bc9bcffe046a59616d6f614dbde9c81"
server_key_iv: "7b8b964f7d784e2d7ae7e0fda2cd7994"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.535416 32198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 18:38:31.536509 32213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.536705 32198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:31.536804 32198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "78030cf7c3464dd1aef8cbbd0d5420da"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-3jql"
server_key: "9bc9bcffe046a59616d6f614dbde9c81"
server_key_iv: "7b8b964f7d784e2d7ae7e0fda2cd7994"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.536892 32198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:31.554965 32198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:31.556895 32198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:31.557080 32198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:31.561026 32198 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:33101
I20260620 18:38:31.561053 32265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:33101 every 8 connection(s)
I20260620 18:38:31.561373 32198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:31.562136 32266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:31.563731 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32198
I20260620 18:38:31.563851 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:31.564024 10677 external_mini_cluster.cc:1428] Setting key b1e396d5ca6c8fbc3cfcdc3ef1f4b6ab
I20260620 18:38:31.564741 32266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da: Bootstrap starting.
I20260620 18:38:31.565320 32266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:31.565665 32266 log.cc:824] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:31.566354 32266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da: No bootstrap required, opened a new log
I20260620 18:38:31.568168 32266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } }
I20260620 18:38:31.568318 32266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:31.568380 32266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78030cf7c3464dd1aef8cbbd0d5420da, State: Initialized, Role: FOLLOWER
I20260620 18:38:31.568531 32266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } }
I20260620 18:38:31.568625 32266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:31.568670 32266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:31.568717 32266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:31.569204 32266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } }
I20260620 18:38:31.569315 32266 leader_election.cc:302] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78030cf7c3464dd1aef8cbbd0d5420da; no voters:
I20260620 18:38:31.569504 32266 leader_election.cc:288] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:31.569545 32271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:31.569662 32271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [term 1 LEADER]: Becoming Leader. State: Replica: 78030cf7c3464dd1aef8cbbd0d5420da, State: Running, Role: LEADER
I20260620 18:38:31.569762 32266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:31.569792 32271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } }
I20260620 18:38:31.570094 32272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } } }
I20260620 18:38:31.570183 32272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:31.570082 32271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78030cf7c3464dd1aef8cbbd0d5420da. Latest consensus state: current_term: 1 leader_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78030cf7c3464dd1aef8cbbd0d5420da" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 33101 } } }
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:31.570477 32271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:31.571035 32277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:31.571547 32277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:31.572866 32277 catalog_manager.cc:1347] Generated new cluster ID: 45c819259d284ae897727b6b8a3ae044
I20260620 18:38:31.572917 32277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:31.581526 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.564509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58213 (local address 127.10.109.126:33101)
0620 18:38:31.564682 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.564691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.564723 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:38:31.565172 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.565178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.565227 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.565336 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:31.566113 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.567125 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.567844 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.568237 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.568359 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.570981 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:31.570985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:31.570986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:31.570998 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:31.579905 (+ 8907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.580358 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.580360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.580361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.580454 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.580690 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.580691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.580691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.580954 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:31.581105 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.581229 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.581317 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260620 18:38:31.582372 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:31.593715 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.583749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50259 (local address 127.10.109.126:33101)
0620 18:38:31.583779 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.583781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.586914 (+ 3133us) server_negotiation.cc:407] Connection header received
0620 18:38:31.586971 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.586972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.586981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.587049 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:31.587684 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.588396 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.589423 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.589727 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.589787 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.591086 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:31.591087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:31.591087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:31.591093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:31.592564 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.592969 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.592970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.592970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.593026 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.593246 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.593247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.593247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.593350 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:31.593429 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.593581 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.593638 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:31.604936 32277 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:31.605463 32277 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:31.609532 32277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78030cf7c3464dd1aef8cbbd0d5420da: Generated new TSK 0
I20260620 18:38:31.609706 32277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:31.614620 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32198
W20260620 18:38:31.618825 32183 connection.cc:535] client connection to 127.10.109.126:33101 recv error: Network error: failed to read from TLS socket (remote: 127.10.109.126:33101): Connection reset by peer (error 104)
2026-06-20T18:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32297](info): set up 2 sockets
Jun 20 18:38:31 dist-test-slave-3jql krb5kdc[32297](info): commencing operation
krb5kdc: starting...
W20260620 18:38:32.655218 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:32 dist-test-slave-3jql krb5kdc[32297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:32.773245 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41113
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:41113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:32.854127 32310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:32.854403 32310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:32.854446 32310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:32.855762 32310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:32.855842 32310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:32.855870 32310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:32.855901 32310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:32.855935 32310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:32.857627 32310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:41113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:32.857942 32310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:32.858208 32310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:32.863572 32318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:32.863657 32316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:32.863986 32310 server_base.cc:1034] running on GCE node
W20260620 18:38:32.864192 32315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:32.864542 32310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:32.864883 32310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:32.866099 32310 hybrid_clock.cc:648] HybridClock initialized: now 1781980712866047 us; error 73 us; skew 500 ppm
Jun 20 18:38:32 dist-test-slave-3jql krb5kdc[32297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980712, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:32.873008 32310 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:32.873521 32310 webserver.cc:460] Webserver started at http://127.10.109.126:39957/ using document root <none> and password file <none>
I20260620 18:38:32.873785 32310 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:32.873854 32310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:32.874003 32310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:32.875252 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "91c13afd714e4548a0de360efe639115"
format_stamp: "Formatted at 2026-06-20 18:38:32 on dist-test-slave-3jql"
server_key: "22d70d54864268caedbeaeb1206f29fe"
server_key_iv: "6574f3321c9c9e334df48ec87c816611"
server_key_version: "encryptionkey@0"
I20260620 18:38:32.875635 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "91c13afd714e4548a0de360efe639115"
format_stamp: "Formatted at 2026-06-20 18:38:32 on dist-test-slave-3jql"
server_key: "22d70d54864268caedbeaeb1206f29fe"
server_key_iv: "6574f3321c9c9e334df48ec87c816611"
server_key_version: "encryptionkey@0"
I20260620 18:38:32.877049 32310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:32.877861 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:32.878237 32310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:32.878343 32310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "91c13afd714e4548a0de360efe639115"
format_stamp: "Formatted at 2026-06-20 18:38:32 on dist-test-slave-3jql"
server_key: "22d70d54864268caedbeaeb1206f29fe"
server_key_iv: "6574f3321c9c9e334df48ec87c816611"
server_key_version: "encryptionkey@0"
I20260620 18:38:32.878430 32310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:32.902310 32310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:32.904366 32310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:32.904536 32310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:32.908730 32310 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:41113
I20260620 18:38:32.908767 32377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:41113 every 8 connection(s)
I20260620 18:38:32.909140 32310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:32.910175 32378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:32.912487 32378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115: Bootstrap starting.
I20260620 18:38:32.913059 32378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:32.913332 32378 log.cc:824] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:32.914332 32378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115: No bootstrap required, opened a new log
I20260620 18:38:32.916604 32378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } }
I20260620 18:38:32.916859 32378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:32.916884 32378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c13afd714e4548a0de360efe639115, State: Initialized, Role: FOLLOWER
I20260620 18:38:32.917081 32378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } }
I20260620 18:38:32.917148 32378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:32.917171 32378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:32.917232 32378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:32.917637 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32310
I20260620 18:38:32.917765 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:32.917963 10677 external_mini_cluster.cc:1428] Setting key 08fd277eac6842e0c794849b0a4503d4
I20260620 18:38:32.918371 32378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } }
I20260620 18:38:32.918484 32378 leader_election.cc:302] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91c13afd714e4548a0de360efe639115; no voters:
I20260620 18:38:32.918655 32378 leader_election.cc:288] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:32.918691 32383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:32.918883 32383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [term 1 LEADER]: Becoming Leader. State: Replica: 91c13afd714e4548a0de360efe639115, State: Running, Role: LEADER
I20260620 18:38:32.919003 32378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:32.919004 32383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } }
I20260620 18:38:32.919492 32385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c13afd714e4548a0de360efe639115. Latest consensus state: current_term: 1 leader_uuid: "91c13afd714e4548a0de360efe639115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } } }
I20260620 18:38:32.919499 32384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91c13afd714e4548a0de360efe639115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c13afd714e4548a0de360efe639115" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 41113 } } }
I20260620 18:38:32.919615 32385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:32.919626 32384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:32.920678 32391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:32.921307 32391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:32.923002 32391 catalog_manager.cc:1347] Generated new cluster ID: 15936ce3435947c891cdad058a8c42f6
I20260620 18:38:32.923105 32391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:32.923975 32382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.918476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46513 (local address 127.10.109.126:41113)
0620 18:38:32.918680 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.918690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.918770 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:38:32.918898 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.918928 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.919001 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.919376 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:32.919476 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.921004 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.921855 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.922439 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.922548 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.923321 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:32.923325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:32.923325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:32.923521 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:32.923555 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:32.923568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:32.923642 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.923759 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.923785 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260620 18:38:32.924952 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:32.930059 32382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.926348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43737 (local address 127.10.109.126:41113)
0620 18:38:32.926385 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.926386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.926528 (+ 142us) server_negotiation.cc:407] Connection header received
0620 18:38:32.926613 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.926614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.926622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.926648 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:32.926835 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.927843 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.928794 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.929102 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.929136 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.929820 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:32.929821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:32.929822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:32.929826 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:32.929837 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:32.929841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:32.929868 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.929982 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.929993 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:38:32.943974 32391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:32.944497 32391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:32.954300 32391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c13afd714e4548a0de360efe639115: Generated new TSK 0
I20260620 18:38:32.954555 32391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:32.959115 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32310
W20260620 18:38:32.963460 32295 connection.cc:535] client connection to 127.10.109.126:41113 recv error: Network error: failed to read from TLS socket (remote: 127.10.109.126:41113): Connection reset by peer (error 104)
2026-06-20T18:38:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:33 dist-test-slave-3jql krb5kdc[32409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:33 dist-test-slave-3jql krb5kdc[32409](info): set up 2 sockets
Jun 20 18:38:33 dist-test-slave-3jql krb5kdc[32409](info): commencing operation
krb5kdc: starting...
W20260620 18:38:33.990875 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:34 dist-test-slave-3jql krb5kdc[32409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:34.113389 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36047
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:34.193538 32422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:34.193723 32422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:34.193746 32422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:34.194926 32422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:34.194969 32422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:34.194983 32422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:34.194995 32422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:34.195008 32422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:34.196372 32422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:36047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:34.196550 32422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:34.196705 32422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:34.201630 32428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.201644 32430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.201704 32427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:34.201705 32422 server_base.cc:1034] running on GCE node
I20260620 18:38:34.202067 32422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:34.202316 32422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:34.203471 32422 hybrid_clock.cc:648] HybridClock initialized: now 1781980714203426 us; error 60 us; skew 500 ppm
Jun 20 18:38:34 dist-test-slave-3jql krb5kdc[32409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980714, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:34.206192 32422 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:34.206794 32422 webserver.cc:460] Webserver started at http://127.10.109.126:35803/ using document root <none> and password file <none>
I20260620 18:38:34.207053 32422 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:34.207129 32422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:34.207291 32422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:34.208344 32422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "5bf79a37be7a4d108293222fc229217a"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-3jql"
server_key: "c05a6ad5442cdf31bf21ec4cc4e45721"
server_key_iv: "be9a1cbca653c9e17f58e239bc3dac47"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.208732 32422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "5bf79a37be7a4d108293222fc229217a"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-3jql"
server_key: "c05a6ad5442cdf31bf21ec4cc4e45721"
server_key_iv: "be9a1cbca653c9e17f58e239bc3dac47"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.210211 32422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:38:34.211196 32437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.211409 32422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:34.211524 32422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "5bf79a37be7a4d108293222fc229217a"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-3jql"
server_key: "c05a6ad5442cdf31bf21ec4cc4e45721"
server_key_iv: "be9a1cbca653c9e17f58e239bc3dac47"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.211606 32422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:34.214989 32422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:34.217176 32422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:34.217338 32422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:34.221195 32422 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:36047
I20260620 18:38:34.221227 32489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:36047 every 8 connection(s)
I20260620 18:38:34.221561 32422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:34.222618 32490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:34.224783 32490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a: Bootstrap starting.
I20260620 18:38:34.225312 32490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:34.225601 32490 log.cc:824] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:34.226315 32490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a: No bootstrap required, opened a new log
I20260620 18:38:34.227970 32490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } }
I20260620 18:38:34.228101 32490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:34.228160 32490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf79a37be7a4d108293222fc229217a, State: Initialized, Role: FOLLOWER
I20260620 18:38:34.228302 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32422
I20260620 18:38:34.228303 32490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } }
I20260620 18:38:34.228384 32490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:34.228420 32490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:34.228434 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:34.228472 32490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:34.228631 10677 external_mini_cluster.cc:1428] Setting key ea7040ff6e06f51b950bc666eece7d0b
I20260620 18:38:34.229565 32490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } }
I20260620 18:38:34.229686 32490 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf79a37be7a4d108293222fc229217a; no voters:
I20260620 18:38:34.229852 32490 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:34.229919 32495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:34.230113 32490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:34.230118 32495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [term 1 LEADER]: Becoming Leader. State: Replica: 5bf79a37be7a4d108293222fc229217a, State: Running, Role: LEADER
I20260620 18:38:34.230316 32495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } }
I20260620 18:38:34.230727 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf79a37be7a4d108293222fc229217a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } } }
I20260620 18:38:34.230777 32497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf79a37be7a4d108293222fc229217a. Latest consensus state: current_term: 1 leader_uuid: "5bf79a37be7a4d108293222fc229217a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf79a37be7a4d108293222fc229217a" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 36047 } } }
I20260620 18:38:34.230882 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:34.230794 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.229115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43459 (local address 127.10.109.126:36047)
0620 18:38:34.229291 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.229305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.229329 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:34.229415 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.229420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.229441 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.229555 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.230275 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.230279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.230279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.230283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.230330 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.230365 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.230421 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.230517 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.230569 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260620 18:38:34.230921 32497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:34.231377 32499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:34.231894 32499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:34.231957 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:34.233395 32499 catalog_manager.cc:1347] Generated new cluster ID: db18e5bd88d84bb994c5dd5ed0370d50
I20260620 18:38:34.233449 32499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:34.234668 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.233558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37479 (local address 127.10.109.126:36047)
0620 18:38:34.233596 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.233598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.233756 (+ 158us) server_negotiation.cc:407] Connection header received
0620 18:38:34.233831 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.233832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.233837 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.233881 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.234447 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.234448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.234448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.234452 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.234465 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.234469 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.234519 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.234573 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.234587 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:38:34.244899 32499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:34.245106 32499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:34.253343 32499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf79a37be7a4d108293222fc229217a: Generated new TSK 0
I20260620 18:38:34.253525 32499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:34.257591 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32422
2026-06-20T18:38:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:34 dist-test-slave-3jql krb5kdc[32521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:34 dist-test-slave-3jql krb5kdc[32521](info): set up 2 sockets
Jun 20 18:38:34 dist-test-slave-3jql krb5kdc[32521](info): commencing operation
krb5kdc: starting...
W20260620 18:38:35.282799 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:35.397636 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:37313
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:35.478257 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:35.478438 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:35.478461 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:35.479534 32534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:35.479571 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:35.479585 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:35.479597 32534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:35.479609 32534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:35.480978 32534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:35.481200 32534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:35.481398 32534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:35.486079 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:35.486405 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:35.486408 32534 server_base.cc:1034] running on GCE node
W20260620 18:38:35.486990 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:35.487251 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:35.487480 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:35.488626 32534 hybrid_clock.cc:648] HybridClock initialized: now 1781980715488614 us; error 27 us; skew 500 ppm
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980715, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:35.491021 32534 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:35.491499 32534 webserver.cc:460] Webserver started at http://127.10.109.126:45271/ using document root <none> and password file <none>
I20260620 18:38:35.491677 32534 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:35.491716 32534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:35.491798 32534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:35.492738 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "79243a667b684074b820b1ac0e1a59a2"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-3jql"
server_key: "806a1c41ca47f1842f3766910ba73213"
server_key_iv: "5a0edf3f330bbed76b679f552755cfd1"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.493029 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "79243a667b684074b820b1ac0e1a59a2"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-3jql"
server_key: "806a1c41ca47f1842f3766910ba73213"
server_key_iv: "5a0edf3f330bbed76b679f552755cfd1"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.494225 32534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:35.494947 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:35.495148 32534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:35.495249 32534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "79243a667b684074b820b1ac0e1a59a2"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-3jql"
server_key: "806a1c41ca47f1842f3766910ba73213"
server_key_iv: "5a0edf3f330bbed76b679f552755cfd1"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.495327 32534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:35.512785 32534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:35.515851 32534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:35.516052 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:35.520874 32534 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:37313
I20260620 18:38:35.520931 32601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:37313 every 8 connection(s)
I20260620 18:38:35.521279 32534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:35.521806 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32534
I20260620 18:38:35.521924 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:35.522117 10677 external_mini_cluster.cc:1428] Setting key aa40366be06ddbae051d4cbb218d1839
I20260620 18:38:35.522367 32602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:35.525731 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2: Bootstrap starting.
I20260620 18:38:35.526394 32602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:35.526703 32602 log.cc:824] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:35.527537 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2: No bootstrap required, opened a new log
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:35.529871 32602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } }
I20260620 18:38:35.530027 32602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:35.530090 32602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79243a667b684074b820b1ac0e1a59a2, State: Initialized, Role: FOLLOWER
I20260620 18:38:35.530248 32602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } }
I20260620 18:38:35.530334 32602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:35.530380 32602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:35.530429 32602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:35.530977 32602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } }
I20260620 18:38:35.531100 32602 leader_election.cc:302] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79243a667b684074b820b1ac0e1a59a2; no voters:
I20260620 18:38:35.531255 32602 leader_election.cc:288] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:35.531298 32607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:35.531471 32607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [term 1 LEADER]: Becoming Leader. State: Replica: 79243a667b684074b820b1ac0e1a59a2, State: Running, Role: LEADER
I20260620 18:38:35.531498 32602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:35.531642 32607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } }
I20260620 18:38:35.531993 32607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79243a667b684074b820b1ac0e1a59a2. Latest consensus state: current_term: 1 leader_uuid: "79243a667b684074b820b1ac0e1a59a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } } }
I20260620 18:38:35.532001 32608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79243a667b684074b820b1ac0e1a59a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79243a667b684074b820b1ac0e1a59a2" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 37313 } } }
I20260620 18:38:35.532114 32607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:35.532143 32608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:35.532325 32614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:35.532866 32614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:35.533640 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.522656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51143 (local address 127.10.109.126:37313)
0620 18:38:35.522841 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.522849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.522882 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:38:35.523586 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.523593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.523639 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.523750 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.524515 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:35.525361 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:35.526474 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:35.527035 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:35.527152 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:35.529709 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:35.529713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:35.529714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:35.529724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:35.531679 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:35.532091 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:35.532094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:35.532094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:35.532202 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:35.532449 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:35.532450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:35.532450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:35.532741 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:35.533239 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:35.533309 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:38:35.533448 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260620 18:38:35.534556 32614 catalog_manager.cc:1347] Generated new cluster ID: 8a65dae58bf1475da6079ad3071b3056
I20260620 18:38:35.534607 32614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:35.534652 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:35.545595 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.536067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35221 (local address 127.10.109.126:37313)
0620 18:38:35.536095 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.536097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.538974 (+ 2877us) server_negotiation.cc:407] Connection header received
0620 18:38:35.539072 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.539074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.539083 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.539150 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.539729 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:35.540447 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:35.541380 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:35.541697 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:35.541772 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:35.542995 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:35.542996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:35.542997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:35.543003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:35.544521 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:35.544881 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:35.544882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:35.544883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:35.544940 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:35.545125 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:35.545126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:35.545126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:35.545248 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:35.545337 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:35.545441 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:38:35.545512 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 18:38:35.551489 32614 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:35.552017 32614 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:35.560820 32614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79243a667b684074b820b1ac0e1a59a2: Generated new TSK 0
I20260620 18:38:35.560990 32614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:35.565757 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32534
2026-06-20T18:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32633](info): set up 2 sockets
Jun 20 18:38:35 dist-test-slave-3jql krb5kdc[32633](info): commencing operation
krb5kdc: starting...
W20260620 18:38:36.598951 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:36 dist-test-slave-3jql krb5kdc[32633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:36.714566 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42341
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:36.792327 32646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:36.792513 32646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:36.792536 32646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:36.793638 32646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:36.793676 32646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:36.793691 32646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:36.793704 32646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:36.793717 32646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:36.795167 32646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:42341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:36.795388 32646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:36.795634 32646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:36.800684 32654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:36.800710 32646 server_base.cc:1034] running on GCE node
W20260620 18:38:36.800686 32651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:36.800712 32652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:36.801159 32646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:36.801425 32646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:36.802568 32646 hybrid_clock.cc:648] HybridClock initialized: now 1781980716802557 us; error 31 us; skew 500 ppm
Jun 20 18:38:36 dist-test-slave-3jql krb5kdc[32633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:36.805147 32646 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:36.805608 32646 webserver.cc:460] Webserver started at http://127.10.109.126:35761/ using document root <none> and password file <none>
I20260620 18:38:36.805835 32646 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:36.805876 32646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:36.806026 32646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:36.807087 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "4968b496c30e4503932447fdd83fcdc9"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-3jql"
server_key: "c77f6c001c68e32565ce5945ada6cc29"
server_key_iv: "9155a1411b0e8bb2fd5d5d0c2ab0274a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.807415 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "4968b496c30e4503932447fdd83fcdc9"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-3jql"
server_key: "c77f6c001c68e32565ce5945ada6cc29"
server_key_iv: "9155a1411b0e8bb2fd5d5d0c2ab0274a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.808696 32646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:36.809484 32661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:36.809692 32646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:36.809790 32646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "4968b496c30e4503932447fdd83fcdc9"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-3jql"
server_key: "c77f6c001c68e32565ce5945ada6cc29"
server_key_iv: "9155a1411b0e8bb2fd5d5d0c2ab0274a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.809908 32646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:36.821156 32646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:36.823668 32646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:36.823812 32646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:36.827442 32646 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:42341
I20260620 18:38:36.827471 32713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:42341 every 8 connection(s)
I20260620 18:38:36.827807 32646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:36.828536 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32646
I20260620 18:38:36.828662 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:36.828846 10677 external_mini_cluster.cc:1428] Setting key ed55462a3642c90f4fe4736f878ce603
I20260620 18:38:36.828855 32714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:36.832188 32714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: Bootstrap starting.
I20260620 18:38:36.832820 32714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:36.833113 32714 log.cc:824] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:36.833871 32714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: No bootstrap required, opened a new log
I20260620 18:38:36.834872 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.829379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45475 (local address 127.10.109.126:42341)
0620 18:38:36.829557 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.829567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.829599 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:38:36.829994 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.830002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.830061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.830204 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:36.830372 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.831735 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.832618 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.833236 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.833353 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.834224 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:36.834227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:36.834227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:36.834379 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:36.834407 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:36.834417 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:36.834512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:36.834567 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 18:38:36.834632 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 18:38:36.835556 32714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } }
I20260620 18:38:36.835716 32714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:36.835755 32714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4968b496c30e4503932447fdd83fcdc9, State: Initialized, Role: FOLLOWER
I20260620 18:38:36.835894 32714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } }
I20260620 18:38:36.835963 32714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:36.836015 32714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:36.836071 32714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:36.836179 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:36.836632 32714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } }
I20260620 18:38:36.836748 32714 leader_election.cc:302] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4968b496c30e4503932447fdd83fcdc9; no voters:
I20260620 18:38:36.836920 32714 leader_election.cc:288] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:36.836956 32719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:36.837138 32719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4968b496c30e4503932447fdd83fcdc9, State: Running, Role: LEADER
I20260620 18:38:36.837162 32714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:36.837347 32719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } }
I20260620 18:38:36.838464 32720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4968b496c30e4503932447fdd83fcdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } } }
I20260620 18:38:36.838558 32720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.838903 32722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4968b496c30e4503932447fdd83fcdc9. Latest consensus state: current_term: 1 leader_uuid: "4968b496c30e4503932447fdd83fcdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4968b496c30e4503932447fdd83fcdc9" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 42341 } } }
I20260620 18:38:36.838968 32722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:36.839051 32739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:36.839155 32739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:36.839255 32740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:36.839432 32740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:36.840813 32740 catalog_manager.cc:1347] Generated new cluster ID: 4e2fae81f6ab43a3a1445fdc522440f1
I20260620 18:38:36.840864 32740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:36.841125 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.837560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51661 (local address 127.10.109.126:42341)
0620 18:38:36.837594 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.837596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.837780 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:38:36.837887 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.837888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.837899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.837934 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:36.838169 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.838948 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.839847 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.840128 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.840197 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.840816 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:36.840817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:36.840817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:36.840821 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:36.840834 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:36.840840 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:36.841045 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:36.841058 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 18:38:36.841076 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:36.862969 32740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:36.863538 32740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:36.872592 32740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4968b496c30e4503932447fdd83fcdc9: Generated new TSK 0
I20260620 18:38:36.872777 32740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:36.879087 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32646
2026-06-20T18:38:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:36 dist-test-slave-3jql krb5kdc[32745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:36 dist-test-slave-3jql krb5kdc[32745](info): set up 2 sockets
Jun 20 18:38:36 dist-test-slave-3jql krb5kdc[32745](info): commencing operation
krb5kdc: starting...
W20260620 18:38:37.919121 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:37 dist-test-slave-3jql krb5kdc[32745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:38.037345 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46523
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:38.118192 32758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:38.118377 32758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:38.118399 32758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:38.119469 32758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:38.119503 32758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:38.119516 32758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:38.119529 32758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:38.119541 32758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:38.120954 32758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:46523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:38.121129 32758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:38.121281 32758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:38.125882 32766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.125914 32764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.126006 32763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:38.126223 32758 server_base.cc:1034] running on GCE node
I20260620 18:38:38.126462 32758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:38.126731 32758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:38.127931 32758 hybrid_clock.cc:648] HybridClock initialized: now 1781980718127910 us; error 35 us; skew 500 ppm
Jun 20 18:38:38 dist-test-slave-3jql krb5kdc[32745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:38.135818 32758 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:38.136279 32758 webserver.cc:460] Webserver started at http://127.10.109.126:40435/ using document root <none> and password file <none>
I20260620 18:38:38.136498 32758 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:38.136543 32758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:38.136677 32758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:38.137691 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "786c8f55cf1e461db8633e631972a5b0"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-3jql"
server_key: "7ad239b69d5c6d226780a587baea4062"
server_key_iv: "01b142aeea21fd12612cdc04c72c3dd3"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.138038 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "786c8f55cf1e461db8633e631972a5b0"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-3jql"
server_key: "7ad239b69d5c6d226780a587baea4062"
server_key_iv: "01b142aeea21fd12612cdc04c72c3dd3"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.139405 32758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:38.140352 305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:38.140744 32758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:38.140839 32758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "786c8f55cf1e461db8633e631972a5b0"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-3jql"
server_key: "7ad239b69d5c6d226780a587baea4062"
server_key_iv: "01b142aeea21fd12612cdc04c72c3dd3"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.140930 32758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:38.144387 32758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:38.146587 32758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:38.146778 32758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:38.150624 32758 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:46523
I20260620 18:38:38.150677 357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:46523 every 8 connection(s)
I20260620 18:38:38.150974 32758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:38.151391 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 32758
I20260620 18:38:38.151505 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:38.151666 10677 external_mini_cluster.cc:1428] Setting key 50f8139cb77647084daa8fad90c06a48
I20260620 18:38:38.152164 358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:38.154052 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.152171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42837 (local address 127.10.109.126:46523)
0620 18:38:38.152335 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:38.152344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:38.152388 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:38:38.152880 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:38.152888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:38.152920 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:38.153067 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:38.153641 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:38.153645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:38.153646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:38.153651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:38.153690 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:38.153703 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:38.153779 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:38.153831 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:38:38.153869 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260620 18:38:38.154989 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:38.155467 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0: Bootstrap starting.
I20260620 18:38:38.156126 358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:38.156484 358 log.cc:824] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:38.157277 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0: No bootstrap required, opened a new log
I20260620 18:38:38.157382 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.156492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60857 (local address 127.10.109.126:46523)
0620 18:38:38.156529 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:38.156531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:38.156542 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:38.156561 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:38.156562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:38.156566 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:38.156633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:38.157155 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:38.157157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:38.157157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:38.157161 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:38.157173 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:38.157177 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:38.157243 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:38.157277 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 18:38:38.157289 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:38.159368 358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } }
I20260620 18:38:38.159489 358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:38.159512 358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786c8f55cf1e461db8633e631972a5b0, State: Initialized, Role: FOLLOWER
I20260620 18:38:38.159621 358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } }
I20260620 18:38:38.159708 358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:38.159730 358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:38.159757 358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:38.160204 358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } }
I20260620 18:38:38.160284 358 leader_election.cc:302] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 786c8f55cf1e461db8633e631972a5b0; no voters:
I20260620 18:38:38.160394 358 leader_election.cc:288] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:38.160471 368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:38.160620 358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:38.160609 368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [term 1 LEADER]: Becoming Leader. State: Replica: 786c8f55cf1e461db8633e631972a5b0, State: Running, Role: LEADER
I20260620 18:38:38.160744 368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } }
I20260620 18:38:38.161114 369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "786c8f55cf1e461db8633e631972a5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } } }
I20260620 18:38:38.161219 369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:38.161128 370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786c8f55cf1e461db8633e631972a5b0. Latest consensus state: current_term: 1 leader_uuid: "786c8f55cf1e461db8633e631972a5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786c8f55cf1e461db8633e631972a5b0" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 46523 } } }
I20260620 18:38:38.161299 370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:38.161502 377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:38.162021 377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:38.163446 377 catalog_manager.cc:1347] Generated new cluster ID: 79fec23ff5c44b67b55368705a181be1
I20260620 18:38:38.163506 377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:38.182638 377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:38.182801 377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:38.192826 377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 786c8f55cf1e461db8633e631972a5b0: Generated new TSK 0
I20260620 18:38:38.192978 377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:38.204682 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 32758
2026-06-20T18:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1325 ms)
[----------] 18 tests from AuthTokenIssuingTest (23805 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:38 dist-test-slave-3jql krb5kdc[389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:38 dist-test-slave-3jql krb5kdc[389](info): set up 2 sockets
Jun 20 18:38:38 dist-test-slave-3jql krb5kdc[389](info): commencing operation
krb5kdc: starting...
W20260620 18:38:39.242944 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:39.357805 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:38559
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:39.437011 402 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:39.437196 402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:39.437215 402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:39.437234 402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:39.438292 402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:39.438328 402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:39.438341 402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:39.438354 402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:39.438365 402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:39.439743 402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:39.439920 402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:39.440068 402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:39.444878 408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.444836 410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.445111 402 server_base.cc:1034] running on GCE node
W20260620 18:38:39.444847 407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.445403 402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:39.445643 402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:39.446827 402 hybrid_clock.cc:648] HybridClock initialized: now 1781980719446810 us; error 33 us; skew 500 ppm
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:39.449237 402 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:39.449640 402 webserver.cc:460] Webserver started at http://127.10.109.126:40263/ using document root <none> and password file <none>
I20260620 18:38:39.449867 402 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:39.449934 402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:39.450085 402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:39.451200 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6e5fe8a03a57558599fa4d61667c80d"
server_key_iv: "a706df2c17b51c31b9bd31958cb7c560"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.451550 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6e5fe8a03a57558599fa4d61667c80d"
server_key_iv: "a706df2c17b51c31b9bd31958cb7c560"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.452931 402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:39.453687 417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.453860 402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:39.453959 402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6e5fe8a03a57558599fa4d61667c80d"
server_key_iv: "a706df2c17b51c31b9bd31958cb7c560"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.454085 402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:39.488209 402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:39.491290 402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:39.491465 402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:39.497846 402 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:38559
I20260620 18:38:39.497903 469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:38559 every 8 connection(s)
I20260620 18:38:39.498358 402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:39.499359 470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:39.501456 470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.502660 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 402
I20260620 18:38:39.502794 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:39.502976 10677 external_mini_cluster.cc:1428] Setting key dccfd4a0298f5f7273b58efc3c4de227
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.109.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.125@KRBTEST.COM" created.
I20260620 18:38:39.515185 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.502268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:38559 (local address 127.0.0.1:46720)
0620 18:38:39.502477 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.502520 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.503217 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.503557 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.503561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.503955 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.504188 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.504194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.505767 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.505768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.506486 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.506495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.507275 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.507276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.507363 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.507959 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.507970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.511213 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.513220 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.513226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.513227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.513476 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.513782 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.513782 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.513783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.513871 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.514344 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.514385 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.514504 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.514619 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.514836 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260620 18:38:39.515193 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.502337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.10.109.126:38559)
0620 18:38:39.502486 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.502495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.502631 (+ 136us) server_negotiation.cc:407] Connection header received
0620 18:38:39.503397 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.503400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.503423 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.503521 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.504301 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.505669 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.506583 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.507166 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.507303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.511297 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.511304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.511304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.511311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.513130 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.513604 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.513607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.513607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.513708 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.513932 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.513933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.513933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.514258 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.514345 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.514668 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.514923 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260620 18:38:39.516577 470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.519052 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.518908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47822)
0620 18:38:39.518965 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.519022 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:38:39.519205 418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
I20260620 18:38:39.519659 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.519552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47838)
0620 18:38:39.519597 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.519627 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:38:39.519845 470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:44747: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
I20260620 18:38:39.533548 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:44747
--webserver_interface=127.10.109.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:39.569092 470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } attempt: 1
I20260620 18:38:39.569567 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.569438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47852)
0620 18:38:39.569497 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.569535 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:38:39.569934 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.569838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47868)
0620 18:38:39.569885 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.569911 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:38:39.570019 470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:44747: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
W20260620 18:38:39.612869 477 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:39.613116 477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:39.613145 477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:39.613186 477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:39.614308 477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:39.614372 477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:39.614396 477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:39.614427 477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:39.614454 477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:39.615896 477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:44747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.109.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:39.616217 477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:39.616436 477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:39.621307 483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.621315 485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.621315 482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.621555 477 server_base.cc:1034] running on GCE node
I20260620 18:38:39.621789 477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:39.622007 477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:39.623173 477 hybrid_clock.cc:648] HybridClock initialized: now 1781980719623163 us; error 40 us; skew 500 ppm
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:39.625710 477 init.cc:377] Logged in from keytab as kudu/127.10.109.125@KRBTEST.COM (short username kudu)
I20260620 18:38:39.626164 477 webserver.cc:460] Webserver started at http://127.10.109.125:39981/ using document root <none> and password file <none>
I20260620 18:38:39.626389 477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:39.626456 477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:39.626601 477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:39.627671 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data/instance:
uuid: "526157ae1d35481e99eff7862c8d60c8"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6c6d42eca58cf0969ceed9d52c194ad"
server_key_iv: "4a7a01e324df2c861a6b4a31b6103ba9"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.628078 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal/instance:
uuid: "526157ae1d35481e99eff7862c8d60c8"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6c6d42eca58cf0969ceed9d52c194ad"
server_key_iv: "4a7a01e324df2c861a6b4a31b6103ba9"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.629364 477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:39.630087 492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.630302 477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:39.630401 477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "526157ae1d35481e99eff7862c8d60c8"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "f6c6d42eca58cf0969ceed9d52c194ad"
server_key_iv: "4a7a01e324df2c861a6b4a31b6103ba9"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.630486 477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:39.638202 470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } attempt: 2
I20260620 18:38:39.638584 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.638455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47872)
0620 18:38:39.638510 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.638559 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:38:39.638947 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.638848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47888)
0620 18:38:39.638895 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.638926 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:38:39.639050 470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:44747: Network error: Client connection negotiation failed: client connection to 127.10.109.125:44747: connect: Connection refused (error 111)
I20260620 18:38:39.645421 477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:39.647730 477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:39.647890 477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:39.651859 477 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.125:44747
I20260620 18:38:39.651895 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.125:44747 every 8 connection(s)
I20260620 18:38:39.652240 477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
I20260620 18:38:39.653064 545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:39.654495 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.658313 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 477
I20260620 18:38:39.658403 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-1/wal/instance
I20260620 18:38:39.658609 10677 external_mini_cluster.cc:1428] Setting key dcecfe04e072e52343e4c7b778ebbe87
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:39.663781 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.655141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:38559 (local address 127.0.0.1:46724)
0620 18:38:39.655305 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.655349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.655821 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.656075 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.656079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.656342 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.656516 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.656520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.657271 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.657272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.658050 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.658052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.658497 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.658499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.658614 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.659239 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.659247 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.660521 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.662417 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.662426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.662428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.662659 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.662925 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.662926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.662926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.663022 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.663331 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.663335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.663443 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.663525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.663635 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260620 18:38:39.663802 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.655170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.10.109.126:38559)
0620 18:38:39.655252 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.655254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.655428 (+ 174us) server_negotiation.cc:407] Connection header received
0620 18:38:39.655934 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.655935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.655948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.656004 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.656620 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.657174 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.658157 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.658389 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.658452 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.660638 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.660640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.660640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.660647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.662293 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.662760 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.662761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.662762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.662828 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.663095 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.663096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.663096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.663247 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.663328 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.663576 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.663667 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:38:39.664487 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.109.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.124@KRBTEST.COM" created.
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:39.673373 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.664750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47894)
0620 18:38:39.664798 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.664809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.664856 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.665118 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.665118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.665222 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.665271 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.665272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.666583 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.666584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.667122 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.667123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.667720 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.667721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.667784 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.668188 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.668192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.670603 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.672343 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.672344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.672344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.672508 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.672722 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.672723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.672724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.672775 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.673051 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.673052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.673088 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.673159 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.673251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:38:39.673439 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.664822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.10.109.125:44747)
0620 18:38:39.664967 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.664971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.664990 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:39.665012 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.665014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.665030 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.665111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.665342 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.666489 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.667204 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.667653 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.667713 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.670703 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.670710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.670711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.670717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.672240 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.672599 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.672601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.672602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.672655 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.672847 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.672848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.672849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.672978 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.673072 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.673188 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.673302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260620 18:38:39.673959 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.674279 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.674184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51634)
0620 18:38:39.674222 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.674261 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 18:38:39.674438 493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
I20260620 18:38:39.674876 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.674802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51648)
0620 18:38:39.674831 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.674860 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 18:38:39.674999 545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:39267: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
I20260620 18:38:39.685578 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:39267
--webserver_interface=127.10.109.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:39.724231 545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } attempt: 1
I20260620 18:38:39.724771 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.724636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51664)
0620 18:38:39.724702 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.724742 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:38:39.725142 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.725064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51666)
0620 18:38:39.725103 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.725127 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 18:38:39.725240 545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:39267: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
I20260620 18:38:39.730206 470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } attempt: 3
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:39.737592 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.730507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47908)
0620 18:38:39.730567 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.730580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.730631 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.730822 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.730823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.730973 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.731044 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.731046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.731968 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.731969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.732545 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.732547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.733070 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.733071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.733131 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.733592 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.733596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.734926 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.736641 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.736642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.736642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.736818 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.737019 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.737020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.737020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.737086 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.737346 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.737347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.737388 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.737442 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.737523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:38:39.737641 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.730605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.10.109.125:44747)
0620 18:38:39.730661 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.730663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.730673 (+ 10us) server_negotiation.cc:407] Connection header received
0620 18:38:39.730699 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.730700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.730710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.730808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.731115 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.731881 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.732621 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.732963 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.733032 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.735003 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.735004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.735004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.735012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.736553 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.736892 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.736893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.736893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.736952 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.737146 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.737147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.737147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.737280 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.737346 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.737480 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.737557 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:39.738384 470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.738754 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.738628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51668)
0620 18:38:39.738680 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.738713 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:38:39.739080 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.738981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51674)
0620 18:38:39.739031 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.739058 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 18:38:39.739171 470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:39267: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
W20260620 18:38:39.768390 552 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:39.768599 552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:39.768616 552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:39.768635 552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:39.769786 552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:39.769825 552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:39.769840 552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:39.769857 552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:39.769869 552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:39.770321 470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } attempt: 1
I20260620 18:38:39.770779 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.770620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51678)
0620 18:38:39.770689 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.770731 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:38:39.771198 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.771073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51682)
0620 18:38:39.771142 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.771173 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:38:39.771332 470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:39267: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
I20260620 18:38:39.771699 552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:38559,127.10.109.125:44747,127.10.109.124:39267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.109.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:39.771889 552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:39.772047 552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:39.776907 558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.776899 557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.777220 560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.777243 552 server_base.cc:1034] running on GCE node
I20260620 18:38:39.777474 552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:39.777758 552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:39.778947 552 hybrid_clock.cc:648] HybridClock initialized: now 1781980719778923 us; error 35 us; skew 500 ppm
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:39.781364 552 init.cc:377] Logged in from keytab as kudu/127.10.109.124@KRBTEST.COM (short username kudu)
I20260620 18:38:39.781847 552 webserver.cc:460] Webserver started at http://127.10.109.124:35611/ using document root <none> and password file <none>
I20260620 18:38:39.782068 552 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:39.782133 552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:39.782282 552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:39.783351 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data/instance:
uuid: "f0a2f3269f6440dea2a1f362bdc5718f"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "b16802f83177b09b8d78d303ebfa29e7"
server_key_iv: "f5b433e88892bbc620641a45ba7e3ea5"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.783695 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal/instance:
uuid: "f0a2f3269f6440dea2a1f362bdc5718f"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "b16802f83177b09b8d78d303ebfa29e7"
server_key_iv: "f5b433e88892bbc620641a45ba7e3ea5"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.784991 552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:38:39.785737 567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.785910 552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 18:38:39.786005 552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "f0a2f3269f6440dea2a1f362bdc5718f"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-3jql"
server_key: "b16802f83177b09b8d78d303ebfa29e7"
server_key_iv: "f5b433e88892bbc620641a45ba7e3ea5"
server_key_version: "encryptionkey@0"
I20260620 18:38:39.786083 552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:39.793393 545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } attempt: 2
I20260620 18:38:39.793757 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.793637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51688)
0620 18:38:39.793694 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.793723 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:38:39.794108 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.794016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51692)
0620 18:38:39.794060 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.794088 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 18:38:39.794199 545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:39267: Network error: Client connection negotiation failed: client connection to 127.10.109.124:39267: connect: Connection refused (error 111)
I20260620 18:38:39.796454 552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:39.798569 552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:39.798722 552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:39.802841 552 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.124:39267
I20260620 18:38:39.802878 619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.124:39267 every 8 connection(s)
I20260620 18:38:39.803213 552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
I20260620 18:38:39.804039 620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:39.805552 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:39.810009 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 552
I20260620 18:38:39.810149 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980452801806-10677-0/minicluster-data/master-2/wal/instance
I20260620 18:38:39.810351 10677 external_mini_cluster.cc:1428] Setting key 9b4228d21b5d9ab1a752f929c1d003cd
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:39.815461 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.806287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.10.109.126:38559)
0620 18:38:39.806351 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.806352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.806582 (+ 230us) server_negotiation.cc:407] Connection header received
0620 18:38:39.807099 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.807100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.807109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.807171 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.807749 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.808225 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.809174 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.809357 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.809418 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.812369 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.812371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.812371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.812379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.813960 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.814375 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.814376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.814376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.814429 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.814694 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.814695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.814695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.814896 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.814965 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.815243 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.815348 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:38:39.815578 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.806255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:38559 (local address 127.0.0.1:46730)
0620 18:38:39.806435 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.806475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.806994 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.807240 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.807243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.807489 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.807666 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.807669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.808310 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.808311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.809092 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.809094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.809594 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.809595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.809747 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.810415 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.810425 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.812258 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.814047 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.814054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.814055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.814265 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.814487 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.814487 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.814488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.814622 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.814968 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.814984 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.815091 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.815215 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.815376 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260620 18:38:39.816159 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:39.820859 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.810795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.10.109.124:39267)
0620 18:38:39.810942 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.810947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.810967 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:38:39.811097 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.811103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.811127 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.811197 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.811993 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.813227 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.814343 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.814810 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.814882 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.817903 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.817910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.817911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.817918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.819639 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.820017 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.820020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.820020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.820080 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.820309 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.820310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.820310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.820447 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.820622 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.820664 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.820743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260620 18:38:39.823767 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.816414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:44747 (local address 127.0.0.1:47910)
0620 18:38:39.816445 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.816455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.816493 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.816688 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.816688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.816822 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.816904 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.816905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.817941 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.817943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.818846 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.818850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.819444 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.819445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.819518 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.819900 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.819905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.821110 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.822705 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.822706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.822707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.822939 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.823168 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.823169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.823169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.823232 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.823451 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.823452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.823493 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.823587 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.823677 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 18:38:39.823770 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.816529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.10.109.125:44747)
0620 18:38:39.816575 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.816577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.816585 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:39.816608 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.816609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.816618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.816685 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.817074 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.817771 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.819120 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.819343 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.819442 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.821184 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.821186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.821186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.821194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.822614 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.823043 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.823044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.823044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.823093 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.823286 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.823286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.823286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.823380 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.823455 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.823613 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.823686 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:39.824342 620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:39.829411 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.821678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.10.109.126:38559)
0620 18:38:39.821724 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.821726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.821734 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:39.821755 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.821756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.821765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.821821 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.822543 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.823281 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.823991 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.824246 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.824300 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.826452 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.826454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.826454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.826461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.828294 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.828647 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.828648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.828649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.828718 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.828914 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.828914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.828915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.829053 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.829221 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.829288 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.829353 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:38:39.831629 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.824576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51714)
0620 18:38:39.824622 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.824632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.824671 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.824781 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.824782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.824904 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.824928 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.824929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.825996 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.825996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.826496 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.826498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.826964 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.826964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.827012 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.827348 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.827351 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.828574 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.830533 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.830534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.830535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.830707 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.830981 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.830982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.830982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.831040 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.831333 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.831334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.831369 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.831444 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.831531 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:38:39.831955 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.824623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.10.109.124:39267)
0620 18:38:39.824653 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.824655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.824662 (+ 7us) server_negotiation.cc:407] Connection header received
0620 18:38:39.824724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.824724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.824732 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.824769 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.824999 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.825443 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.826587 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.826884 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.826963 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.828676 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.828677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.828678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.828685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.830441 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.830835 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.830835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.830836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.830890 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.831117 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.831117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.831117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.831236 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.831308 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.831560 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.831634 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:39.833891 620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: Bootstrap starting.
I20260620 18:38:39.835036 620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:39.835326 620 log.cc:824] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:39.836110 620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: No bootstrap required, opened a new log
I20260620 18:38:39.837620 620 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.837778 620 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2f3269f6440dea2a1f362bdc5718f, State: Initialized, Role: FOLLOWER
I20260620 18:38:39.837873 620 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.838150 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.838203 620 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:39.838248 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.838598 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.830352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.10.109.125:44747)
0620 18:38:39.830385 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.830387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.830436 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:38:39.830505 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.830506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.830516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.830570 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.831153 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.831851 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.832847 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.833108 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.833150 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.835885 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.835886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.835887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.835893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.837366 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.837748 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.837749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.837749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.837826 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.838057 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.838058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.838058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.838176 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.838377 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.838419 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.838496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 18:38:39.840000 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:39.840054 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:39.840062 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:39.841087 444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46742
I20260620 18:38:39.840874 552 master_runner.cc:186] Error getting master registration for 127.10.109.126:38559: OK, instance_id { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" instance_seqno: 1781980719496250 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:39.843228 552 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:39.846472 470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } attempt: 2
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:39.854036 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.846723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51722)
0620 18:38:39.846787 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.846798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.846840 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.847019 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.847020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.847191 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.847239 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.847241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.848103 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.848104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.848894 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.848895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.849359 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.849360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.849416 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.849827 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.849832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.851238 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.852935 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.852936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.852937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.853119 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.853347 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.853348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.853349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.853412 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.853704 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.853705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.853750 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.853842 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.853940 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:38:39.854035 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.846823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.10.109.124:39267)
0620 18:38:39.846881 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.846883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.846891 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:39.846930 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.846930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.846940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.847001 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.847327 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.848024 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.848979 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.849281 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.849352 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.851330 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.851331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.851332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.851340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.852825 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.853212 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.853213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.853213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.853276 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.853498 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.853499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.853499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.853629 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.853711 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.853880 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.853952 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:38:39.855875 470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: Bootstrap starting.
I20260620 18:38:39.856597 470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:39.856925 470 log.cc:824] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:39.857733 470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: No bootstrap required, opened a new log
I20260620 18:38:39.859223 470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.859371 470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1ec32863c24e6fbefe5b5c0cf2856c, State: Initialized, Role: FOLLOWER
I20260620 18:38:39.859462 470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.859784 470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:39.859831 444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:38:39.859877 444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:39.861020 444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
W20260620 18:38:39.861711 651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 18:38:39.861261 444 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [CANDIDATE]: Term 1 election: Requested vote from peers 526157ae1d35481e99eff7862c8d60c8 (127.10.109.125:44747), f0a2f3269f6440dea2a1f362bdc5718f (127.10.109.124:39267)
W20260620 18:38:39.861768 651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:39.859763 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.861915 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.862206 594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f0a2f3269f6440dea2a1f362bdc5718f"
I20260620 18:38:39.862357 594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:39.863420 594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1ec32863c24e6fbefe5b5c0cf2856c in term 1.
I20260620 18:38:39.863648 418 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1ec32863c24e6fbefe5b5c0cf2856c, f0a2f3269f6440dea2a1f362bdc5718f; no voters:
I20260620 18:38:39.864128 640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:39.864275 640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [term 1 LEADER]: Becoming Leader. State: Replica: 8d1ec32863c24e6fbefe5b5c0cf2856c, State: Running, Role: LEADER
I20260620 18:38:39.864367 640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.864780 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1ec32863c24e6fbefe5b5c0cf2856c. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.864871 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:39.865211 660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:39.863461 402 master_runner.cc:186] Error getting master registration for 127.10.109.125:44747: OK, instance_id { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" instance_seqno: 1781980719650726 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:39.865667 402 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:39.865640 660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:39.867614 594 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [term 1 FOLLOWER]: Refusing update from remote peer 8d1ec32863c24e6fbefe5b5c0cf2856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:39.867964 658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:39.870023 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1ec32863c24e6fbefe5b5c0cf2856c. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.870116 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.871050 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.871124 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:39.871223 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.871304 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.871443 662 mvcc.cc:205] Tried to move back new op lower bound from 7298993028575678464 to 7298993028565204992. Current Snapshot: MvccSnapshot[applied={T|T < 7298993028575678464}]
I20260620 18:38:39.884338 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.862943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.10.109.126:38559)
0620 18:38:39.862988 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.862990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.874887 (+ 11897us) server_negotiation.cc:407] Connection header received
0620 18:38:39.874939 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.874940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.874952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.875011 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.876101 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.876839 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.878366 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.878662 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.878705 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.880969 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.880970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.880971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.880988 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.882685 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.883290 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.883291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.883291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.883357 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.883640 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.883641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.883641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.883774 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.883870 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.884174 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.884247 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 18:38:39.885007 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.874972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.10.109.125:44747)
0620 18:38:39.875036 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.875038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.875317 (+ 279us) server_negotiation.cc:407] Connection header received
0620 18:38:39.875480 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.875481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.875506 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.875538 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.876684 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.877393 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.878965 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.879275 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.879315 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.881407 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.881408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.881409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.881427 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.883348 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.883928 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.883930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.883930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.883996 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.884294 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.884295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.884305 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.884425 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.884505 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.884827 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.884917 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:39.885316 545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } attempt: 3
I20260620 18:38:39.885854 660 catalog_manager.cc:1347] Generated new cluster ID: 81a8be2dac33424f87acde2ff786fe0e
I20260620 18:38:39.885902 660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:39.893138 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.875326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.10.109.124:39267)
0620 18:38:39.875361 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.875363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.884936 (+ 9573us) server_negotiation.cc:407] Connection header received
0620 18:38:39.885269 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.885270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.885292 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.885320 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.886385 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.887099 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.888365 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.888652 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.888686 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.890131 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.890132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.890132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.890138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.891776 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.892281 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.892282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.892283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.892354 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.892570 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.892571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.892571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.892691 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.892784 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.892982 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.893055 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:39.896185 660 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:39.896868 660 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 18:38:39 dist-test-slave-3jql krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:39.901750 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.885517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:39267 (local address 127.0.0.1:51746)
0620 18:38:39.885601 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.885611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.885646 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.893287 (+ 7641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.893287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.893443 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.893482 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.893484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.894367 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.894368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.895169 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.895172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.896349 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.896350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.896409 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.896912 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.896917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.898302 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.900670 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.900671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.900671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.900883 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.901125 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.901126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.901126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.901205 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.901472 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.901473 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.901521 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.901578 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.901660 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:38:39.901782 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.885617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.10.109.124:39267)
0620 18:38:39.893192 (+ 7575us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.893194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.893200 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:38:39.893214 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.893215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.893223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.893272 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.893591 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.894290 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.895235 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.896010 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.896297 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.898397 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.898398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.898398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.898404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.900581 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.900971 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.900972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.900972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.901026 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.901270 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.901270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.901271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.901396 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.901464 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.901620 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.901693 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7563,"thread_start_us":89,"threads_started":1}
I20260620 18:38:39.903488 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: Bootstrap starting.
I20260620 18:38:39.904132 545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:39.904381 545 log.cc:824] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:39.905193 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: No bootstrap required, opened a new log
I20260620 18:38:39.906972 545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.907094 545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526157ae1d35481e99eff7862c8d60c8, State: Initialized, Role: FOLLOWER
I20260620 18:38:39.907155 545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } }
I20260620 18:38:39.907421 545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:39.907397 670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.907485 670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.909760 660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c: Generated new TSK 0
I20260620 18:38:39.909942 660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:39.911536 518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:39.912446 518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [term 1 FOLLOWER]: Refusing update from remote peer 8d1ec32863c24e6fbefe5b5c0cf2856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:39.912921 658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:39.912926 519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "526157ae1d35481e99eff7862c8d60c8"
I20260620 18:38:39.914027 519 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1ec32863c24e6fbefe5b5c0cf2856c in term 1.
I20260620 18:38:39.915431 670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1ec32863c24e6fbefe5b5c0cf2856c. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.915501 670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.915619 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:39.915683 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:39.919313 681 catalog_manager.cc:1259] Loaded cluster ID: 81a8be2dac33424f87acde2ff786fe0e
I20260620 18:38:39.919420 681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: loading cluster ID for follower catalog manager: success
I20260620 18:38:39.920629 681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: acquiring CA information for follower catalog manager: success
I20260620 18:38:39.921015 681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 526157ae1d35481e99eff7862c8d60c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:38:39.929940 549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.928466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.10.109.125:44747)
0620 18:38:39.928516 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.928519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.929343 (+ 824us) server_negotiation.cc:407] Connection header received
0620 18:38:39.929440 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.929440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.929450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.929479 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.929881 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47938: BlockingRecv error: recv got EOF from 127.0.0.1:47938 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 18:38:39.930691 473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.928697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.10.109.126:38559)
0620 18:38:39.928732 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.928734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.930189 (+ 1455us) server_negotiation.cc:407] Connection header received
0620 18:38:39.930282 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.930283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.930293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.930320 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.930626 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46770: BlockingRecv error: recv got EOF from 127.0.0.1:46770 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 18:38:39.931380 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.928993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.10.109.124:39267)
0620 18:38:39.929030 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.929032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.930919 (+ 1887us) server_negotiation.cc:407] Connection header received
0620 18:38:39.931013 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.931014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.931023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.931053 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.931326 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51762: BlockingRecv error: recv got EOF from 127.0.0.1:51762 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:39.936362 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.932827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.10.109.125:44747)
0620 18:38:39.932863 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.932864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.933127 (+ 263us) server_negotiation.cc:407] Connection header received
0620 18:38:39.933238 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.933239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.933267 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.933307 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:39.933478 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.934282 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.935460 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.935761 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.935800 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.936006 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:39.936117 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:39.936164 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.936275 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.936290 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 18:38:39.939502 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.938734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.10.109.124:39267)
0620 18:38:39.938794 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.938795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.939013 (+ 218us) server_negotiation.cc:407] Connection header received
0620 18:38:39.939112 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.939113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.939128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.939156 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.939437 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51764: BlockingRecv error: recv got EOF from 127.0.0.1:51764 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:39.944864 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.941693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.10.109.125:44747)
0620 18:38:39.941722 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.941724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.941946 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:38:39.942036 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.942037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.942045 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.942071 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:39.942225 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.942976 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.944067 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.944353 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.944406 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.944628 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:39.944658 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:39.944684 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.944786 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.944802 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 18:38:39.947693 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.947006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.10.109.124:39267)
0620 18:38:39.947040 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.947042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.947226 (+ 184us) server_negotiation.cc:407] Connection header received
0620 18:38:39.947319 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.947320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.947329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.947356 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.947650 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51768: BlockingRecv error: recv got EOF from 127.0.0.1:51768 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:38:39.953680 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.950717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.10.109.125:44747)
0620 18:38:39.950767 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.950768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.950996 (+ 228us) server_negotiation.cc:407] Connection header received
0620 18:38:39.951083 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.951084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.951092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.951117 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:39.951259 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.951950 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.952952 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.953207 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.953253 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.953438 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:39.953466 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:39.953490 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.953599 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.953613 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 18:38:39.956457 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.955715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.10.109.124:39267)
0620 18:38:39.955747 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.955749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.955976 (+ 227us) server_negotiation.cc:407] Connection header received
0620 18:38:39.956086 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.956087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.956096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.956123 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.956418 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51778: BlockingRecv error: recv got EOF from 127.0.0.1:51778 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:39.964659 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.961569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.10.109.125:44747)
0620 18:38:39.961601 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.961603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.961818 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:38:39.961913 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.961914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.961923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.961949 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:39.962110 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.962862 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.963898 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.964180 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.964218 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.964399 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:39.964427 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:39.964453 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.964582 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.964599 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:38:39.971423 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.966942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.10.109.124:39267)
0620 18:38:39.966974 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.966976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.970942 (+ 3966us) server_negotiation.cc:407] Connection header received
0620 18:38:39.971039 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.971040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.971050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.971078 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.971380 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51782: BlockingRecv error: recv got EOF from 127.0.0.1:51782 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:38:39.983613 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.980547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.10.109.125:44747)
0620 18:38:39.980580 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.980582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.980780 (+ 198us) server_negotiation.cc:407] Connection header received
0620 18:38:39.980874 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.980874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.980883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.980911 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:39.981073 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.981768 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.982844 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.983121 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.983163 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.983355 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:39.983386 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:39.983414 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.983532 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.983549 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:38:39.986346 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:39.985680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.10.109.124:39267)
0620 18:38:39.985714 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.985716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.985875 (+ 159us) server_negotiation.cc:407] Connection header received
0620 18:38:39.985966 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.985967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.985977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.986005 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.986303 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51788: BlockingRecv error: recv got EOF from 127.0.0.1:51788 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:38:40.006862 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.003625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.10.109.125:44747)
0620 18:38:40.003671 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.003673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.003895 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:38:40.004001 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.004002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.004012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.004041 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:40.004220 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.004959 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.006024 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.006279 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.006320 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.006523 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:40.006559 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:40.006589 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.006714 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.006775 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:38:40.009837 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:40.009082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.10.109.124:39267)
0620 18:38:40.009120 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.009122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.009296 (+ 174us) server_negotiation.cc:407] Connection header received
0620 18:38:40.009391 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.009392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.009416 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.009446 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.009785 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51794: BlockingRecv error: recv got EOF from 127.0.0.1:51794 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 18:38:40.045759 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.043171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.10.109.125:44747)
0620 18:38:40.043219 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.043221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.043365 (+ 144us) server_negotiation.cc:407] Connection header received
0620 18:38:40.043441 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.043442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.043454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.043525 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:40.043651 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.044222 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.045076 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.045273 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.045345 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.045513 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:40.045539 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:40.045597 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.045648 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.045668 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 18:38:40.048130 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:40.047610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.10.109.124:39267)
0620 18:38:40.047655 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.047657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.047665 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:40.047688 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.047689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.047698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.047766 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.048076 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51802: BlockingRecv error: recv got EOF from 127.0.0.1:51802 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:38:40.115769 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.113465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.10.109.125:44747)
0620 18:38:40.113544 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.113546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.113628 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:38:40.113691 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.113692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.113702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.113749 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:40.113916 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.114395 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.115236 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.115412 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.115465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.115582 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:40.115608 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:40.115646 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.115697 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.115711 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 18:38:40.118016 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:40.117488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.10.109.124:39267)
0620 18:38:40.117552 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.117554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.117569 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:38:40.117638 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.117639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.117648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.117695 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.117961 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51812: BlockingRecv error: recv got EOF from 127.0.0.1:51812 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:40.249516 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.247261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.10.109.125:44747)
0620 18:38:40.247329 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.247331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.247367 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:38:40.247447 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.247448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.247458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.247513 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:40.247666 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.248139 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.248899 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.249076 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.249127 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.249277 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:40.249302 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:40.249339 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.249392 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.249406 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 18:38:40.251852 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:40.251351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.10.109.124:39267)
0620 18:38:40.251419 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.251422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.251436 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:38:40.251500 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.251501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.251510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.251561 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.251812 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51822: BlockingRecv error: recv got EOF from 127.0.0.1:51822 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:40.511559 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.509118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.10.109.125:44747)
0620 18:38:40.509198 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.509201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.509212 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:40.509265 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.509266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.509276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.509341 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:40.509496 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.509979 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.510760 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.511041 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.511125 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.511293 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:40.511326 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:40.511394 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.511447 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.511469 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 18:38:40.513808 624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:40.513335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.10.109.124:39267)
0620 18:38:40.513391 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.513392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.513403 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:38:40.513433 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.513434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.513442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.513490 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.513738 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47816: BlockingRecv error: recv got EOF from 127.0.0.1:47816 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:40.536669 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1ec32863c24e6fbefe5b5c0cf2856c" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 38559 } } peers { permanent_uuid: "526157ae1d35481e99eff7862c8d60c8" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 44747 } } peers { permanent_uuid: "f0a2f3269f6440dea2a1f362bdc5718f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 39267 } } }
I20260620 18:38:40.536825 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ec32863c24e6fbefe5b5c0cf2856c [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:40.840480 637 catalog_manager.cc:1259] Loaded cluster ID: 81a8be2dac33424f87acde2ff786fe0e
I20260620 18:38:40.840579 637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: loading cluster ID for follower catalog manager: success
I20260620 18:38:40.841418 637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: acquiring CA information for follower catalog manager: success
I20260620 18:38:40.841715 637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0a2f3269f6440dea2a1f362bdc5718f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:41.029301 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.026986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.10.109.125:44747)
0620 18:38:41.027105 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.027107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.027153 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:41.027214 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.027215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.027225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.027284 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:41.027410 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.027910 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.028628 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.028856 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.028920 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.029043 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:41.029072 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:41.029113 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.029160 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.029176 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260620 18:38:41.033188 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.030960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.10.109.124:39267)
0620 18:38:41.031094 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.031095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.031103 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:41.031143 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.031144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.031190 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.031250 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:41.031319 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.031767 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.032395 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.032594 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.032695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.032841 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:41.032936 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:41.032988 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.033046 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.033083 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260620 18:38:41.037523 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.034796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.10.109.125:44747)
0620 18:38:41.034834 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.034836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.034945 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:38:41.035032 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.035033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.035044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.035095 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:41.035181 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.035897 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.036758 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.036980 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.037057 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.037255 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:41.037285 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:41.037311 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.037422 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.037445 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:41.037989 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.034947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.10.109.124:39267)
0620 18:38:41.034981 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.034982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.035301 (+ 319us) server_negotiation.cc:407] Connection header received
0620 18:38:41.035379 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.035380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.035389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.035438 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:41.035548 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.035976 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.037205 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.037391 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.037475 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.037723 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:41.037753 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:41.037782 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.037887 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.037909 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 18:38:41.038055 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.034780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.10.109.126:38559)
0620 18:38:41.035003 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.035005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.035013 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:41.035053 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.035053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.035092 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.035177 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:41.035313 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.036055 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.037124 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.037436 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.037491 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.037705 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:41.037799 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:41.037890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.037952 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.037974 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":51,"threads_started":1}
I20260620 18:38:41.040033 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 402
I20260620 18:38:41.044718 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 477
I20260620 18:38:41.049337 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 552
2026-06-20T18:38:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2846 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:41 dist-test-slave-3jql krb5kdc[824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:41 dist-test-slave-3jql krb5kdc[824](info): set up 2 sockets
Jun 20 18:38:41 dist-test-slave-3jql krb5kdc[824](info): commencing operation
krb5kdc: starting...
W20260620 18:38:42.087031 10677 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/test-user.keytab.
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.109.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
Entry for principal kudu/127.10.109.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab.
I20260620 18:38:42.198352 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45879
--webserver_interface=127.10.109.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:42.277719 838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:42.277963 838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.277995 838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.278035 838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.279206 838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:42.279273 838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:42.279300 838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.279333 838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:42.279356 838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:42.280768 838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.126.keytab
--principal=kudu/127.10.109.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.126:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.109.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:42.281033 838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.281240 838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:42.285821 843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.285820 844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.285879 838 server_base.cc:1034] running on GCE node
W20260620 18:38:42.285843 846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.286351 838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.286616 838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.287763 838 hybrid_clock.cc:648] HybridClock initialized: now 1781980722287744 us; error 37 us; skew 500 ppm
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.290143 838 init.cc:377] Logged in from keytab as kudu/127.10.109.126@KRBTEST.COM (short username kudu)
I20260620 18:38:42.290591 838 webserver.cc:460] Webserver started at http://127.10.109.126:35783/ using document root <none> and password file <none>
I20260620 18:38:42.290841 838 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.290889 838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:42.291018 838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:42.291993 838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data/instance:
uuid: "7ad708f9c1ec4e6cb4ae94b00997c101"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "2fc0e18a748ae855915ca83af3a717e5"
server_key_iv: "883d45cbf883c399bee9541e7044f307"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.292327 838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance:
uuid: "7ad708f9c1ec4e6cb4ae94b00997c101"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "2fc0e18a748ae855915ca83af3a717e5"
server_key_iv: "883d45cbf883c399bee9541e7044f307"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.293571 838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:42.294217 853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.294380 838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:42.294471 838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal
uuid: "7ad708f9c1ec4e6cb4ae94b00997c101"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "2fc0e18a748ae855915ca83af3a717e5"
server_key_iv: "883d45cbf883c399bee9541e7044f307"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.294548 838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.319185 838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.321401 838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.321575 838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.325819 838 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.126:45879
I20260620 18:38:42.325932 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.126:45879 every 8 connection(s)
I20260620 18:38:42.326231 838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:42.327116 906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:42.328544 906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.332600 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 838
I20260620 18:38:42.332676 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-0/wal/instance
I20260620 18:38:42.332913 10677 external_mini_cluster.cc:1428] Setting key 05eacba05ea0c27fbb768210d98d3dcf
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:42.338305 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.329135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45879 (local address 127.0.0.1:60320)
0620 18:38:42.329307 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.329346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.329768 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.330083 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.330086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.330326 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.330485 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.330489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.331511 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.331512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.332204 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.332206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.332762 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.332763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.332863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.333329 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.333339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.335113 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.336904 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.336909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.336911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.337092 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.337333 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.337334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.337334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.337418 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.337743 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.337748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.337803 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.337914 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.338131 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":92,"thread_start_us":86,"threads_started":1}
I20260620 18:38:42.338312 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.329233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.10.109.126:45879)
0620 18:38:42.329387 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.329392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.329426 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:42.329931 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.329933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.329951 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.330033 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.330576 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.331421 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.332321 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.332621 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.332770 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.335214 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.335220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.335221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.335234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.336803 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.337184 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.337186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.337187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.337262 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.337483 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.337484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.337484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.337672 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.337763 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.337946 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.338075 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260620 18:38:42.339176 906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.339550 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.339398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53528)
0620 18:38:42.339483 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.339528 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 18:38:42.339691 855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
I20260620 18:38:42.340128 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.340030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53542)
0620 18:38:42.340079 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.340104 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:38:42.340227 906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:38251: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.109.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
Entry for principal kudu/127.10.109.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab.
I20260620 18:38:42.356827 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:38251
--webserver_interface=127.10.109.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:42.389456 906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } attempt: 1
I20260620 18:38:42.389935 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.389792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53552)
0620 18:38:42.389872 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.389908 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:38:42.390266 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.390172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53566)
0620 18:38:42.390216 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.390241 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:38:42.390358 906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:38251: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
W20260620 18:38:42.435606 913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:42.435833 913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.435863 913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.435904 913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.437004 913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:42.437065 913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:42.437091 913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.437124 913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:42.437144 913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:42.438584 913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.125.keytab
--principal=kudu/127.10.109.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.125:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.109.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:42.438864 913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.439076 913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:42.443667 918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.443696 919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.443681 921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.443974 913 server_base.cc:1034] running on GCE node
I20260620 18:38:42.444218 913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.444500 913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.445660 913 hybrid_clock.cc:648] HybridClock initialized: now 1781980722445641 us; error 38 us; skew 500 ppm
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.448148 913 init.cc:377] Logged in from keytab as kudu/127.10.109.125@KRBTEST.COM (short username kudu)
I20260620 18:38:42.448609 913 webserver.cc:460] Webserver started at http://127.10.109.125:44443/ using document root <none> and password file <none>
I20260620 18:38:42.448848 913 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.448916 913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:42.449066 913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:42.450074 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data/instance:
uuid: "bc9b03496ba14d4b86996c4fbadfe180"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "1b35db8542e607b85812d4be0c8e1b93"
server_key_iv: "4f039ebe3026c6170084ebc1a3534405"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.450423 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal/instance:
uuid: "bc9b03496ba14d4b86996c4fbadfe180"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "1b35db8542e607b85812d4be0c8e1b93"
server_key_iv: "4f039ebe3026c6170084ebc1a3534405"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.451774 913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:42.452467 928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.452641 913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.452737 913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal
uuid: "bc9b03496ba14d4b86996c4fbadfe180"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "1b35db8542e607b85812d4be0c8e1b93"
server_key_iv: "4f039ebe3026c6170084ebc1a3534405"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.452816 913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.458487 906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } attempt: 2
I20260620 18:38:42.458873 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.458717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53574)
0620 18:38:42.458810 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.458846 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:38:42.459167 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.459075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53580)
0620 18:38:42.459119 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.459142 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:38:42.459249 906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.125:38251: Network error: Client connection negotiation failed: client connection to 127.10.109.125:38251: connect: Connection refused (error 111)
I20260620 18:38:42.467849 913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.469627 913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.469791 913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.473798 913 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.125:38251
I20260620 18:38:42.473831 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.125:38251 every 8 connection(s)
I20260620 18:38:42.474157 913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/data/info.pb
I20260620 18:38:42.475018 981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:42.476444 981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.481067 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 913
I20260620 18:38:42.481175 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-1/wal/instance
I20260620 18:38:42.481384 10677 external_mini_cluster.cc:1428] Setting key 311ff1af68cc2d927238fe9426a431b9
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:42.485033 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.477068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.10.109.126:45879)
0620 18:38:42.477124 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.477126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.477328 (+ 202us) server_negotiation.cc:407] Connection header received
0620 18:38:42.477751 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.477752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.477763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.477812 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.478385 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.478874 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.479797 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.479989 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.480049 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.482220 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.482222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.482222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.482229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.483680 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.484087 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.484088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.484088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.484146 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.484401 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.484402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.484402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.484513 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.484601 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.484860 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.484937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:38:42.485195 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.477067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45879 (local address 127.0.0.1:60328)
0620 18:38:42.477211 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.477243 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.477674 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.477859 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.477862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.478119 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.478300 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.478304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.478945 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.478946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.479719 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.479721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.480081 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.480081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.480175 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.480654 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.480661 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.482120 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.483784 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.483791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.483792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.484010 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.484224 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.484225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.484225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.484328 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.484594 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.484600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.484731 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.484837 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.484978 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260620 18:38:42.485719 981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.109.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.109.124@KRBTEST.COM" created.
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:42.494107 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.485972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53592)
0620 18:38:42.486020 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.486031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.486066 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.486284 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.486285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.486418 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.486462 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.486463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.487379 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.487380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.487856 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.487857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.488364 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.488366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.488446 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.488787 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.488791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.491340 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.493115 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.493116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.493117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.493303 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.493549 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.493550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.493550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.493600 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.493908 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.493908 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.493941 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.493992 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.494052 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:38:42.494272 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.486044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.10.109.125:38251)
0620 18:38:42.486167 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.486170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.486184 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:38:42.486197 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.486199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.486207 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.486284 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.486543 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.487279 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.487951 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.488255 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.488344 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.491435 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.491443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.491443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.491455 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.493035 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.493408 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.493411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.493411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.493476 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.493664 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.493665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.493665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.493842 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.493926 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.494028 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.494121 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260620 18:38:42.494846 981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.495124 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.495042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36030)
0620 18:38:42.495070 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.495107 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 18:38:42.495242 930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
I20260620 18:38:42.495645 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.495557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36034)
0620 18:38:42.495600 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.495623 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:38:42.495743 981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:42713: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
Entry for principal kudu/127.10.109.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab.
I20260620 18:38:42.506435 10677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx7fjtl/build/release/bin/kudu
/tmp/dist-test-taskx7fjtl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:42713
--webserver_interface=127.10.109.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.109.84:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:42.544979 981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } attempt: 1
I20260620 18:38:42.545495 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.545363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36040)
0620 18:38:42.545427 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.545467 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:38:42.545850 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.545772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36050)
0620 18:38:42.545811 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.545834 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:38:42.545939 981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:42713: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
I20260620 18:38:42.550380 906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } attempt: 3
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:42.557200 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.550656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53598)
0620 18:38:42.550713 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.550725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.550808 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.550932 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.550933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.551095 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.551157 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.551159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.551826 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.551827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.552343 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.552345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.552738 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.552739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.552819 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.553268 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.553276 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.554437 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.556214 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.556215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.556215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.556389 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.556629 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.556630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.556630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.556693 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.556971 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.556972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.557010 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.557064 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.557138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:38:42.557251 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.550697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.10.109.125:38251)
0620 18:38:42.550736 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.550754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.550795 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:38:42.550872 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.550872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.550883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.550928 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.551247 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.551741 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.552437 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.552654 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.552740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.554516 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.554517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.554518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.554525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.556144 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.556460 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.556461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.556462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.556522 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.556765 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.556766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.556767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.556895 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.556972 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.557097 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.557171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:42.557612 906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.557898 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.557815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36058)
0620 18:38:42.557846 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.557879 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 18:38:42.558168 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.558116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36072)
0620 18:38:42.558137 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.558156 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 18:38:42.558272 906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:42713: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
W20260620 18:38:42.586341 988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:38:42.586539 988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.586556 988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.586576 988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.587641 988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:42.587674 988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:42.587688 988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.587702 988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:42.587713 988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:42.589052 988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.109.84:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.109.126:45879,127.10.109.125:38251,127.10.109.124:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/krb5kdc/kudu_127.10.109.124.keytab
--principal=kudu/127.10.109.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.109.124:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.109.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:14 UTC on ccycloud.quasar-dtsakl.root.comops.site
build id 4457676
I20260620 18:38:42.589226 988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.589377 988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:38:42.589438 906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } attempt: 1
I20260620 18:38:42.589874 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.589746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36080)
0620 18:38:42.589808 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.589845 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:38:42.590327 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.590224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36082)
0620 18:38:42.590274 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.590309 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:38:42.590441 906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:42713: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
W20260620 18:38:42.594161 994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.594177 996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.594194 993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.594380 988 server_base.cc:1034] running on GCE node
I20260620 18:38:42.594616 988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.594893 988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.596037 988 hybrid_clock.cc:648] HybridClock initialized: now 1781980722596025 us; error 26 us; skew 500 ppm
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.598439 988 init.cc:377] Logged in from keytab as kudu/127.10.109.124@KRBTEST.COM (short username kudu)
I20260620 18:38:42.598910 988 webserver.cc:460] Webserver started at http://127.10.109.124:36549/ using document root <none> and password file <none>
I20260620 18:38:42.599159 988 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.599227 988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:42.599380 988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:42.600405 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data/instance:
uuid: "f163f4d0a52b41b2a3f159e04ba20a8f"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "730972a524926929974bcb6b53d3913d"
server_key_iv: "47bcc0f5080bff14a46f2a0315b9196c"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.600790 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal/instance:
uuid: "f163f4d0a52b41b2a3f159e04ba20a8f"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "730972a524926929974bcb6b53d3913d"
server_key_iv: "47bcc0f5080bff14a46f2a0315b9196c"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.602072 988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 18:38:42.602797 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.602983 988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 18:38:42.603080 988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data,/tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal
uuid: "f163f4d0a52b41b2a3f159e04ba20a8f"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-3jql"
server_key: "730972a524926929974bcb6b53d3913d"
server_key_iv: "47bcc0f5080bff14a46f2a0315b9196c"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.603164 988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.614071 981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } attempt: 2
I20260620 18:38:42.614444 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.614326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36094)
0620 18:38:42.614382 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.614417 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:38:42.614815 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.614700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36098)
0620 18:38:42.614761 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.614787 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 18:38:42.614892 981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.109.124:42713: Network error: Client connection negotiation failed: client connection to 127.10.109.124:42713: connect: Connection refused (error 111)
I20260620 18:38:42.620107 988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.621793 988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.621950 988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.625859 988 rpc_server.cc:307] RPC server started. Bound to: 127.10.109.124:42713
I20260620 18:38:42.625886 1055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.109.124:42713 every 8 connection(s)
I20260620 18:38:42.626207 988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/data/info.pb
I20260620 18:38:42.627110 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:42.628507 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.630901 10677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx7fjtl/build/release/bin/kudu as pid 988
I20260620 18:38:42.630995 10677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx7fjtl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980452801806-10677-0/minicluster-data/master-2/wal/instance
I20260620 18:38:42.631187 10677 external_mini_cluster.cc:1428] Setting key 5923588f0eb84303bd61e14179f9bb17
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:42.637765 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.629142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.10.109.126:45879)
0620 18:38:42.629216 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.629218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.629353 (+ 135us) server_negotiation.cc:407] Connection header received
0620 18:38:42.629777 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.629778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.629787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.629839 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.630375 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.631000 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.632428 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.632706 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.632783 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.634890 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.634891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.634892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.634899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.636368 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.636824 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.636826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.636826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.636886 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.637122 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.637123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.637124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.637234 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.637309 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.637571 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.637647 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:38:42.637964 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.629094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.126:45879 (local address 127.0.0.1:60330)
0620 18:38:42.629239 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.629270 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.629692 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.629884 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.629887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.630107 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.630279 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.630282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.631117 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.631118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.632337 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.632341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.632796 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.632797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.632896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.633370 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.633385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.634806 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.636482 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.636495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.636496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.636736 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.636954 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.636955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.636955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.637055 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.637302 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.637325 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.637445 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.637550 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.637745 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 18:38:42.638510 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.640137 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.631553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.10.109.124:42713)
0620 18:38:42.631687 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.631691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.631716 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:42.631783 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.631785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.631799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.631857 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.632521 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.633293 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.634121 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.634399 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.634489 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.637066 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.637074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.637074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.637083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.639065 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.639399 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.639401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.639401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.639471 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.639656 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.639657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.639657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.639763 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.639873 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.639966 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.640021 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.126@KRBTEST.COM
I20260620 18:38:42.646224 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.638772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.125:38251 (local address 127.0.0.1:53610)
0620 18:38:42.638870 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.638881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.638942 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.639057 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.639058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.639188 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.639222 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.639223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.640086 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.640087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.640981 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.640983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.641455 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.641456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.641495 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.641840 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.641845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.643381 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.645119 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.645120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.645120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.645282 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.645564 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.645564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.645565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.645649 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.645905 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.645906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.645947 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.646048 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.646135 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 18:38:42.646229 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.638824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.10.109.125:38251)
0620 18:38:42.638869 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.638871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.638918 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:38:42.639003 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.639004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.639011 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.639053 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.639297 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.640008 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.641039 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.641378 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.641454 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.643480 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.643481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.643481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.643487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.645015 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.645395 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.645397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.645397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.645457 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.645719 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.645720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.645721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.645837 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.645913 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.646077 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.646146 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:42.646589 1056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:42.648797 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.640903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.10.109.126:45879)
0620 18:38:42.640938 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.640940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.640982 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:38:42.641071 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.641072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.641081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.641114 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.641655 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.642431 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.643428 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.643677 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.643754 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.645971 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.645972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.645973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.645979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.647634 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.647982 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.647984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.647984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.648035 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.648238 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.648239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.648239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.648351 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.648523 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.648565 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.648630 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.124@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:42.652998 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.646855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36122)
0620 18:38:42.646902 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.646912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.646951 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.647117 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.647118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.647245 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.647266 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.647267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.647846 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.647848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.648299 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.648300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.648702 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.648703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.648750 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.649075 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.649080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.650427 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.652043 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.652044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.652044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.652190 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.652401 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.652402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.652403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.652459 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.652704 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.652705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.652741 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.652806 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.652896 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:38:42.653164 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.646902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.10.109.124:42713)
0620 18:38:42.646931 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.646933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.646939 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:38:42.647005 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.647005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.647047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.647115 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.647338 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.647772 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.648391 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.648615 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.648701 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.650526 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.650527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.650527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.650533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.651943 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.652280 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.652281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.652281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.652329 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.652536 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.652537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.652537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.652637 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.652712 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.652835 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.652941 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":17}
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.109.125@KRBTEST.COM
I20260620 18:38:42.654718 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: Bootstrap starting.
I20260620 18:38:42.655786 1056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:42.656111 1056 log.cc:824] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:42.656755 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.649611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.10.109.125:38251)
0620 18:38:42.649644 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.649645 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.649654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:42.649674 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.649674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.649682 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.649734 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.650272 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.650865 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.651508 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.651729 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.651799 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.653963 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.653964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.653964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.653971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.655602 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.655969 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.655971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.655971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.656039 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.656214 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.656216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.656216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.656348 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.656492 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.656562 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.656665 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 18:38:42.657044 1056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: No bootstrap required, opened a new log
I20260620 18:38:42.658079 10677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:42.659132 1056 raft_consensus.cc:348] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.659289 1056 raft_consensus.cc:729] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f163f4d0a52b41b2a3f159e04ba20a8f, State: Initialized, Role: FOLLOWER
I20260620 18:38:42.659366 1056 consensus_queue.cc:260] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.659631 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.659670 1056 sys_catalog.cc:564] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:42.659736 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.659205 880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60342
W20260620 18:38:42.661473 1073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:42.661526 1073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:42.661768 988 master_runner.cc:186] Error getting master registration for 127.10.109.126:45879: OK, instance_id { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" instance_seqno: 1781980722324606 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:42.664748 988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:42.665592 906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } attempt: 2
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.126@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:42.672256 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.665884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.10.109.124:42713)
0620 18:38:42.665924 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.665943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.665951 (+ 8us) server_negotiation.cc:407] Connection header received
0620 18:38:42.665973 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.665974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.665983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.666037 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.666286 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.666729 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.667429 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.667640 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.667715 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.669489 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.669490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.669491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.669497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.671104 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.671486 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.671487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.671488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.671539 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.671742 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.671743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.671744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.671891 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.671954 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.672103 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.672172 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:42.672262 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.665824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36130)
0620 18:38:42.665877 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.665887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.665928 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.666041 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.666041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.666181 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.666210 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.666211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.666843 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.666844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.667331 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.667333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.667713 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.667714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.667769 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.668186 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.668191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.669402 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.671185 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.671186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.671186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.671408 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.671612 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.671613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.671613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.671678 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.671966 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.671967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.672012 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.672085 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.672171 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:38:42.673651 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101: Bootstrap starting.
I20260620 18:38:42.674527 906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:42.674825 906 log.cc:824] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:42.675627 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101: No bootstrap required, opened a new log
I20260620 18:38:42.676903 906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.677032 906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad708f9c1ec4e6cb4ae94b00997c101, State: Initialized, Role: FOLLOWER
I20260620 18:38:42.677105 906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.677390 906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:42.677371 1076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.677403 880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:38:42.677466 1076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.677484 880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:42.678124 880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.678298 880 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [CANDIDATE]: Term 1 election: Requested vote from peers bc9b03496ba14d4b86996c4fbadfe180 (127.10.109.125:38251), f163f4d0a52b41b2a3f159e04ba20a8f (127.10.109.124:42713)
I20260620 18:38:42.679270 1030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f"
I20260620 18:38:42.679385 1030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:42.680164 1030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad708f9c1ec4e6cb4ae94b00997c101 in term 1.
I20260620 18:38:42.680356 855 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad708f9c1ec4e6cb4ae94b00997c101, f163f4d0a52b41b2a3f159e04ba20a8f; no voters:
I20260620 18:38:42.680464 1076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:42.680619 1076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [term 1 LEADER]: Becoming Leader. State: Replica: 7ad708f9c1ec4e6cb4ae94b00997c101, State: Running, Role: LEADER
I20260620 18:38:42.680709 1076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.681094 1076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad708f9c1ec4e6cb4ae94b00997c101. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.681172 1076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:42.681769 1094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:42.682286 1094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:42.682566 838 master_runner.cc:186] Error getting master registration for 127.10.109.125:38251: OK, instance_id { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" instance_seqno: 1781980722472691 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:42.685024 838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:42.688385 1030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [term 1 FOLLOWER]: Refusing update from remote peer 7ad708f9c1ec4e6cb4ae94b00997c101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:42.688660 1091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:42.689116 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.679343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.10.109.126:45879)
0620 18:38:42.679383 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.679385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.680962 (+ 1577us) server_negotiation.cc:407] Connection header received
0620 18:38:42.681038 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.681039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.681049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.681108 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.681990 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.682658 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.683750 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.683950 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.684035 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.685482 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.685483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.685484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.685490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.687388 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.687821 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.687822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.687822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.687878 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.688218 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.688219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.688220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.688349 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.688452 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.688939 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.689033 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 18:38:42.689771 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad708f9c1ec4e6cb4ae94b00997c101. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.689851 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.695837 1091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.695930 1091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:42.695945 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.686977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.10.109.125:38251)
0620 18:38:42.687026 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.687028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.687170 (+ 142us) server_negotiation.cc:407] Connection header received
0620 18:38:42.687272 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.687273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.687283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.687313 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.688868 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.689565 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.690982 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.691275 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.691315 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.692757 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.692758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.692759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.692765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.694364 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.695041 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.695042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.695042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.695097 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.695306 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.695307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.695308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.695434 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.695532 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.695751 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.695828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 18:38:42.696342 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.696424 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.696606 1100 mvcc.cc:205] Tried to move back new op lower bound from 7298993040128937984 to 7298993040100782080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993040128937984}]
I20260620 18:38:42.696760 1097 mvcc.cc:205] Tried to move back new op lower bound from 7298993040128937984 to 7298993040100782080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993040128937984}]
I20260620 18:38:42.699654 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.687210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.10.109.124:42713)
0620 18:38:42.687246 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.687248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.689770 (+ 2522us) server_negotiation.cc:407] Connection header received
0620 18:38:42.690020 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.690021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.690031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.690061 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.691159 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.691850 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.693747 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.694055 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.694100 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.696682 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.696684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.696684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.696691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.698423 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.698856 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.698857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.698857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.698932 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.699149 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.699150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.699150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.699252 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.699359 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.699481 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.699551 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 18:38:42.700649 1094 catalog_manager.cc:1347] Generated new cluster ID: 9dba6234d13e47c88fff5d4fb42026a5
I20260620 18:38:42.700696 1094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:42.706003 981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } attempt: 3
Jun 20 18:38:42 dist-test-slave-3jql krb5kdc[824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.109.125@KRBTEST.COM for kudu/127.10.109.124@KRBTEST.COM
I20260620 18:38:42.714386 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.706195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.109.124:42713 (local address 127.0.0.1:36146)
0620 18:38:42.706257 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.706266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.706294 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.706638 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.706638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.706803 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.706835 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.706836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.707684 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.707685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.708428 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.708429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.708898 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.708898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.708961 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.709321 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.709325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.710612 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.713278 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.713279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.713280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.713463 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.713723 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.713724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.713724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.713785 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.714134 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.714134 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.714182 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.714222 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.714296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:38:42.715056 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.706350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.10.109.124:42713)
0620 18:38:42.706465 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.706467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.706519 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:38:42.706536 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.706537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.706544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.706904 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.706940 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.707594 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.708522 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.708817 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.710701 (+ 1884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.710831 (+ 130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.710832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.710832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.710869 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.713164 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.713546 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.713547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.713547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.713619 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.713859 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.713859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.713860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.714050 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.714705 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.714800 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.714919 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 18:38:42.716599 981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: Bootstrap starting.
I20260620 18:38:42.717211 981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:42.717514 981 log.cc:824] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:42.721886 981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: No bootstrap required, opened a new log
I20260620 18:38:42.722288 1094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:42.722990 1094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:42.724114 981 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.724241 981 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9b03496ba14d4b86996c4fbadfe180, State: Initialized, Role: FOLLOWER
I20260620 18:38:42.724318 981 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } }
I20260620 18:38:42.724627 1105 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.724673 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc9b03496ba14d4b86996c4fbadfe180"
I20260620 18:38:42.724720 954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:42.725109 1105 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.725567 954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [term 1 FOLLOWER]: Refusing update from remote peer 7ad708f9c1ec4e6cb4ae94b00997c101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:42.724649 981 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:42.726349 955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad708f9c1ec4e6cb4ae94b00997c101 in term 1.
I20260620 18:38:42.727036 1091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:42.729377 1105 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.729504 1105 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.729605 1094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101: Generated new TSK 0
I20260620 18:38:42.729794 1094 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:38:42.730618 1116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:42.730758 1116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:42.730614 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad708f9c1ec4e6cb4ae94b00997c101. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.731043 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:42.733214 1076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad708f9c1ec4e6cb4ae94b00997c101" member_type: VOTER last_known_addr { host: "127.10.109.126" port: 45879 } } peers { permanent_uuid: "bc9b03496ba14d4b86996c4fbadfe180" member_type: VOTER last_known_addr { host: "127.10.109.125" port: 38251 } } peers { permanent_uuid: "f163f4d0a52b41b2a3f159e04ba20a8f" member_type: VOTER last_known_addr { host: "127.10.109.124" port: 42713 } } }
I20260620 18:38:42.733317 1076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad708f9c1ec4e6cb4ae94b00997c101 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:42.757562 909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.756133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.10.109.126:45879)
0620 18:38:42.756181 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.756183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.756906 (+ 723us) server_negotiation.cc:407] Connection header received
0620 18:38:42.757014 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.757016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.757028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.757058 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.757488 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60366: BlockingRecv error: recv got EOF from 127.0.0.1:60366 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 18:38:42.758265 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.756569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.10.109.125:38251)
0620 18:38:42.756612 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.756614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.757792 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 18:38:42.757885 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.757886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.757896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.757925 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.758210 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53644: BlockingRecv error: recv got EOF from 127.0.0.1:53644 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 18:38:42.758903 1060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.756795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.10.109.124:42713)
0620 18:38:42.756832 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.756834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.758478 (+ 1644us) server_negotiation.cc:407] Connection header received
0620 18:38:42.758511 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.758512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.758523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.758596 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.758848 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36154: BlockingRecv error: recv got EOF from 127.0.0.1:36154 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 18:38:42.760550 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.759812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.10.109.125:38251)
0620 18:38:42.759843 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.759845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.760083 (+ 238us) server_negotiation.cc:407] Connection header received
0620 18:38:42.760184 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.760185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.760201 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.760229 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.760513 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53658: BlockingRecv error: recv got EOF from 127.0.0.1:53658 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 18:38:42.763707 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.762723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.10.109.125:38251)
0620 18:38:42.762785 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.762787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.763242 (+ 455us) server_negotiation.cc:407] Connection header received
0620 18:38:42.763333 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.763334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.763343 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.763371 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.763664 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53660: BlockingRecv error: recv got EOF from 127.0.0.1:53660 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 18:38:42.771420 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.766825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.10.109.125:38251)
0620 18:38:42.766876 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.766878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.770936 (+ 4058us) server_negotiation.cc:407] Connection header received
0620 18:38:42.771035 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.771036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.771046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.771075 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.771373 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53670: BlockingRecv error: recv got EOF from 127.0.0.1:53670 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 18:38:42.777230 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.776470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.10.109.125:38251)
0620 18:38:42.776506 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.776507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.776734 (+ 227us) server_negotiation.cc:407] Connection header received
0620 18:38:42.776830 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.776831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.776842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.776872 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.777184 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53676: BlockingRecv error: recv got EOF from 127.0.0.1:53676 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 18:38:42.787245 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.786440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.10.109.125:38251)
0620 18:38:42.786485 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.786487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.786658 (+ 171us) server_negotiation.cc:407] Connection header received
0620 18:38:42.786804 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.786805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.786817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.786848 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.787186 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53678: BlockingRecv error: recv got EOF from 127.0.0.1:53678 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 18:38:42.805337 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.804450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.10.109.125:38251)
0620 18:38:42.804501 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.804503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.804779 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:38:42.804885 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.804886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.804897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.804927 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.805282 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53682: BlockingRecv error: recv got EOF from 127.0.0.1:53682 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 18:38:42.839802 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.838897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.10.109.125:38251)
0620 18:38:42.838943 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.838946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.839209 (+ 263us) server_negotiation.cc:407] Connection header received
0620 18:38:42.839310 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.839311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.839323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.839352 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.839747 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53696: BlockingRecv error: recv got EOF from 127.0.0.1:53696 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 18:38:42.905850 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.905169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.10.109.125:38251)
0620 18:38:42.905238 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.905241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.905340 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:38:42.905428 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.905429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.905439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.905490 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.905787 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53708: BlockingRecv error: recv got EOF from 127.0.0.1:53708 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 18:38:43.036070 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:43.035076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.10.109.125:38251)
0620 18:38:43.035159 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.035162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.035275 (+ 113us) server_negotiation.cc:407] Connection header received
0620 18:38:43.035380 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.035381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.035393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.035486 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.036016 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53720: BlockingRecv error: recv got EOF from 127.0.0.1:53720 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 18:38:43.293900 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:43.293377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.10.109.125:38251)
0620 18:38:43.293451 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.293453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.293462 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:43.293488 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.293489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.293499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.293561 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.293853 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53726: BlockingRecv error: recv got EOF from 127.0.0.1:53726 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:38:43.661926 1073 catalog_manager.cc:1259] Loaded cluster ID: 9dba6234d13e47c88fff5d4fb42026a5
I20260620 18:38:43.661998 1073 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: loading cluster ID for follower catalog manager: success
I20260620 18:38:43.662830 1073 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: acquiring CA information for follower catalog manager: success
I20260620 18:38:43.663184 1073 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f163f4d0a52b41b2a3f159e04ba20a8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:43.731189 1116 catalog_manager.cc:1259] Loaded cluster ID: 9dba6234d13e47c88fff5d4fb42026a5
I20260620 18:38:43.731259 1116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: loading cluster ID for follower catalog manager: success
I20260620 18:38:43.732100 1116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: acquiring CA information for follower catalog manager: success
I20260620 18:38:43.732420 1116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc9b03496ba14d4b86996c4fbadfe180: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:43.809559 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.807151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.10.109.125:38251)
0620 18:38:43.807275 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.807277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.807286 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:43.807312 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.807313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.807361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.807449 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.807581 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.808071 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.808794 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.809015 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.809068 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.809213 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.809314 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.809373 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.809475 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.809493 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260620 18:38:43.813982 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.811457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.10.109.124:42713)
0620 18:38:43.811578 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.811581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.811590 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:43.811634 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.811635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.811680 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.811778 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.811919 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.812407 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.813139 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.813371 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.813449 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.813631 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.813734 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.813819 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.813874 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.813894 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260620 18:38:43.818436 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.815769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.10.109.125:38251)
0620 18:38:43.815834 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.815837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.815929 (+ 92us) server_negotiation.cc:407] Connection header received
0620 18:38:43.816024 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.816025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.816036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.816066 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.816244 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.816728 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.817631 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.817921 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.817995 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.818167 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.818207 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.818265 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.818331 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.818352 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:38:43.818964 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.816068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.10.109.124:42713)
0620 18:38:43.816102 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.816103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.816109 (+ 6us) server_negotiation.cc:407] Connection header received
0620 18:38:43.816129 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.816130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.816140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.816188 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.816326 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.817090 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.818033 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.818308 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.818374 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.818669 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.818693 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.818780 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.818877 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.818896 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 18:38:43.819084 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.815734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.10.109.126:45879)
0620 18:38:43.815889 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.815891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.815900 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:38:43.815967 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.815968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.816018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.816127 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.816263 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.817061 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.817976 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.818288 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.818507 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.818567 (+ 60us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.818677 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.818767 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.818853 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.818984 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260620 18:38:43.826507 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 838
I20260620 18:38:43.831992 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 913
I20260620 18:38:43.837003 10677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx7fjtl/build/release/bin/kudu with pid 988
2026-06-20T18:38:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2788 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5634 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271043 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST